[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:35:52 dist-test-slave-wjdr krb5kdc[7138](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:35:52 dist-test-slave-wjdr krb5kdc[7138](info): set up 2 sockets
Sep 25 22:35:52 dist-test-slave-wjdr krb5kdc[7138](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260925 22:35:53.860996 7131 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Sep 25 22:35:53 dist-test-slave-wjdr krb5kdc[7138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:35:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:35:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.246.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab.
Entry for principal kudu/127.6.246.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab.
I20260925 22:35:53.986123 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:46343
--webserver_interface=127.6.246.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab
--principal=kudu/127.6.246.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:35341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.246.254:46343
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:35:54.106249 7151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:35:54.106582 7151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:35:54.106657 7151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:35:54.110796 7151 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:35:54.110896 7151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:35:54.110937 7151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:35:54.110989 7151 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:35:54.111027 7151 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:35:54.116595 7151 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:35341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.246.254:46343
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab
--principal=kudu/127.6.246.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:46343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.246.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7151
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:35:54.117707 7151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:35:54.118795 7151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:35:54.129287 7159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:35:54.129303 7157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:35:54.129472 7156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:35:54.130278 7151 server_base.cc:1034] running on GCE node
I20260925 22:35:54.130827 7151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:35:54.131978 7151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:35:54.133169 7151 hybrid_clock.cc:648] HybridClock initialized: now 1790375754133078 us; error 107 us; skew 500 ppm
Sep 25 22:35:54 dist-test-slave-wjdr krb5kdc[7138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375754, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:35:54.136502 7151 init.cc:377] Logged in from keytab as kudu/127.6.246.254@KRBTEST.COM (short username kudu)
I20260925 22:35:54.137511 7151 webserver.cc:460] Webserver started at http://127.6.246.254:36871/ using document root <none> and password file <none>
I20260925 22:35:54.138077 7151 fs_manager.cc:359] Metadata directory not provided
I20260925 22:35:54.138166 7151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:35:54.138379 7151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:35:54.140753 7151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/minicluster-data/master-0/data/instance:
uuid: "c6504d3eba9c4dbb850b840da0b433af"
format_stamp: "Formatted at 2026-09-25 22:35:54 on dist-test-slave-wjdr"
server_key: "3a9d537fe8101f991f562896062aca7b"
server_key_iv: "bcfec394e246f5ea1b94112fbc75c76c"
server_key_version: "encryptionkey@0"
I20260925 22:35:54.141631 7151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/minicluster-data/master-0/wal/instance:
uuid: "c6504d3eba9c4dbb850b840da0b433af"
format_stamp: "Formatted at 2026-09-25 22:35:54 on dist-test-slave-wjdr"
server_key: "3a9d537fe8101f991f562896062aca7b"
server_key_iv: "bcfec394e246f5ea1b94112fbc75c76c"
server_key_version: "encryptionkey@0"
I20260925 22:35:54.145440 7151 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:35:54.147814 7166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:35:54.149200 7151 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:35:54.149346 7151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/minicluster-data/master-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/minicluster-data/master-0/wal
uuid: "c6504d3eba9c4dbb850b840da0b433af"
format_stamp: "Formatted at 2026-09-25 22:35:54 on dist-test-slave-wjdr"
server_key: "3a9d537fe8101f991f562896062aca7b"
server_key_iv: "bcfec394e246f5ea1b94112fbc75c76c"
server_key_version: "encryptionkey@0"
I20260925 22:35:54.149507 7151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:35:54.177201 7151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:35:54.180605 7151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:35:54.180837 7151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:35:54.188000 7151 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.254:46343
I20260925 22:35:54.188024 7218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.254:46343 every 8 connection(s)
I20260925 22:35:54.188941 7151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
I20260925 22:35:54.191633 7219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:35:54.193059 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 7151
I20260925 22:35:54.193220 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/minicluster-data/master-0/wal/instance
I20260925 22:35:54.194463 7131 external_mini_cluster.cc:1428] Setting key 10b77955c23a35b3357c02bc2c00e051
I20260925 22:35:54.197770 7219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6504d3eba9c4dbb850b840da0b433af: Bootstrap starting.
I20260925 22:35:54.200138 7219 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6504d3eba9c4dbb850b840da0b433af: Neither blocks nor log segments found. Creating new log.
I20260925 22:35:54.200905 7219 log.cc:824] T 00000000000000000000000000000000 P c6504d3eba9c4dbb850b840da0b433af: Log is configured to *not* fsync() on all Append() calls
I20260925 22:35:54.202639 7219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6504d3eba9c4dbb850b840da0b433af: No bootstrap required, opened a new log
I20260925 22:35:54.205583 7219 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6504d3eba9c4dbb850b840da0b433af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6504d3eba9c4dbb850b840da0b433af" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 46343 } }
I20260925 22:35:54.206018 7219 raft_consensus.cc:374] T 00000000000000000000000000000000 P c6504d3eba9c4dbb850b840da0b433af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:35:54.206111 7219 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6504d3eba9c4dbb850b840da0b433af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6504d3eba9c4dbb850b840da0b433af, State: Initialized, Role: FOLLOWER
I20260925 22:35:54.206601 7219 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6504d3eba9c4dbb850b840da0b433af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6504d3eba9c4dbb850b840da0b433af" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 46343 } }
I20260925 22:35:54.206774 7219 raft_consensus.cc:388] T 00000000000000000000000000000000 P c6504d3eba9c4dbb850b840da0b433af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:35:54.207002 7219 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6504d3eba9c4dbb850b840da0b433af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:35:54.207144 7219 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6504d3eba9c4dbb850b840da0b433af [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:35:54.208046 7219 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6504d3eba9c4dbb850b840da0b433af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6504d3eba9c4dbb850b840da0b433af" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 46343 } }
I20260925 22:35:54.208376 7219 leader_election.cc:302] T 00000000000000000000000000000000 P c6504d3eba9c4dbb850b840da0b433af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6504d3eba9c4dbb850b840da0b433af; no voters:
I20260925 22:35:54.208714 7219 leader_election.cc:288] T 00000000000000000000000000000000 P c6504d3eba9c4dbb850b840da0b433af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:35:54.208793 7224 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6504d3eba9c4dbb850b840da0b433af [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:35:54.209088 7224 raft_consensus.cc:686] T 00000000000000000000000000000000 P c6504d3eba9c4dbb850b840da0b433af [term 1 LEADER]: Becoming Leader. State: Replica: c6504d3eba9c4dbb850b840da0b433af, State: Running, Role: LEADER
I20260925 22:35:54.209386 7224 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6504d3eba9c4dbb850b840da0b433af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6504d3eba9c4dbb850b840da0b433af" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 46343 } }
I20260925 22:35:54.210011 7219 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6504d3eba9c4dbb850b840da0b433af [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:35:54.210861 7225 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6504d3eba9c4dbb850b840da0b433af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c6504d3eba9c4dbb850b840da0b433af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6504d3eba9c4dbb850b840da0b433af" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 46343 } } }
I20260925 22:35:54.210991 7225 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6504d3eba9c4dbb850b840da0b433af [sys.catalog]: This master's current role is: LEADER
Sep 25 22:35:54 dist-test-slave-wjdr krb5kdc[7138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
I20260925 22:35:54.212513 7226 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6504d3eba9c4dbb850b840da0b433af [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6504d3eba9c4dbb850b840da0b433af. Latest consensus state: current_term: 1 leader_uuid: "c6504d3eba9c4dbb850b840da0b433af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6504d3eba9c4dbb850b840da0b433af" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 46343 } } }
I20260925 22:35:54.213950 7226 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6504d3eba9c4dbb850b840da0b433af [sys.catalog]: This master's current role is: LEADER
W20260925 22:35:54.215628 7240 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c6504d3eba9c4dbb850b840da0b433af: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:35:54.215705 7240 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:35:54.215791 7233 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:35:54.216645 7233 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:35:54.219269 7221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:35:54.197387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37538 (local address 127.6.246.254:46343)
0925 22:35:54.197786 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0925 22:35:54.197801 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:35:54.198815 (+ 1014us) server_negotiation.cc:407] Connection header received
0925 22:35:54.201988 (+ 3173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:35:54.202001 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:35:54.202278 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:35:54.202592 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:35:54.204686 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:35:54.206228 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:35:54.207846 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:35:54.208438 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:35:54.208652 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:35:54.213845 (+ 5193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:35:54.213853 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:35:54.213868 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:35:54.213897 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:35:54.215829 (+ 1932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:35:54.216337 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:35:54.216344 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:35:54.216349 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:35:54.216534 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:35:54.216981 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:35:54.216984 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:35:54.216986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:35:54.217354 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:35:54.217565 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0925 22:35:54.218110 (+ 545us) server_negotiation.cc:299] Negotiation successful
0925 22:35:54.218379 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":134,"threads_started":1}
I20260925 22:35:54.222337 7233 catalog_manager.cc:1347] Generated new cluster ID: 74126ba06a1a4a3dbf7910384c9a12ba
I20260925 22:35:54.222445 7233 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:35:54.250993 7233 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:35:54.252385 7233 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:35:54.266825 7233 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c6504d3eba9c4dbb850b840da0b433af: Generated new TSK 0
I20260925 22:35:54.267737 7233 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.246.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.193.keytab.
Entry for principal kudu/127.6.246.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.193.keytab.
I20260925 22:35:54.307626 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.193:0
--local_ip_for_outbound_sockets=127.6.246.193
--webserver_interface=127.6.246.193
--webserver_port=0
--tserver_master_addrs=127.6.246.254:46343
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.193.keytab
--principal=kudu/127.6.246.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:35341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:35:54.417187 7245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:35:54.417433 7245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:35:54.417477 7245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:35:54.421428 7245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:35:54.421479 7245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:35:54.421574 7245 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.193
I20260925 22:35:54.426775 7245 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:35341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.193.keytab
--principal=kudu/127.6.246.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.246.193
--webserver_port=0
--tserver_master_addrs=127.6.246.254:46343
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7245
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.193
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:35:54.427807 7245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:35:54.428833 7245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 22:35:54.438329 7245 server_base.cc:1034] running on GCE node
W20260925 22:35:54.438395 7250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:35:54.438314 7251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:35:54.438347 7253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:35:54.438982 7245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:35:54.439628 7245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:35:54.440863 7245 hybrid_clock.cc:648] HybridClock initialized: now 1790375754440840 us; error 51 us; skew 500 ppm
Sep 25 22:35:54 dist-test-slave-wjdr krb5kdc[7138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375754, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:35:54.443603 7245 init.cc:377] Logged in from keytab as kudu/127.6.246.193@KRBTEST.COM (short username kudu)
I20260925 22:35:54.444558 7245 webserver.cc:460] Webserver started at http://127.6.246.193:44881/ using document root <none> and password file <none>
I20260925 22:35:54.445111 7245 fs_manager.cc:359] Metadata directory not provided
I20260925 22:35:54.445199 7245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:35:54.445444 7245 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:35:54.447455 7245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/minicluster-data/ts-0/data/instance:
uuid: "a2bf5aa00abb4f14b69c5520a25247ae"
format_stamp: "Formatted at 2026-09-25 22:35:54 on dist-test-slave-wjdr"
server_key: "1073634d6db678a7e797b47d8c4a60e1"
server_key_iv: "e5552e7a1eea31958c9662343cbf9aa9"
server_key_version: "encryptionkey@0"
I20260925 22:35:54.448019 7245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/minicluster-data/ts-0/wal/instance:
uuid: "a2bf5aa00abb4f14b69c5520a25247ae"
format_stamp: "Formatted at 2026-09-25 22:35:54 on dist-test-slave-wjdr"
server_key: "1073634d6db678a7e797b47d8c4a60e1"
server_key_iv: "e5552e7a1eea31958c9662343cbf9aa9"
server_key_version: "encryptionkey@0"
I20260925 22:35:54.451640 7245 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 22:35:54.454152 7260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:35:54.455343 7245 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 22:35:54.455485 7245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/minicluster-data/ts-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/minicluster-data/ts-0/wal
uuid: "a2bf5aa00abb4f14b69c5520a25247ae"
format_stamp: "Formatted at 2026-09-25 22:35:54 on dist-test-slave-wjdr"
server_key: "1073634d6db678a7e797b47d8c4a60e1"
server_key_iv: "e5552e7a1eea31958c9662343cbf9aa9"
server_key_version: "encryptionkey@0"
I20260925 22:35:54.455595 7245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:35:54.469954 7245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:35:54.472914 7245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:35:54.473115 7245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:35:54.473743 7245 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:35:54.474880 7245 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:35:54.474949 7245 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:35:54.475015 7245 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:35:54.475045 7245 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:35:54.483630 7245 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.193:39995
I20260925 22:35:54.483677 7373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.193:39995 every 8 connection(s)
I20260925 22:35:54.484583 7245 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
I20260925 22:35:54.493906 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 7245
I20260925 22:35:54.494033 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/minicluster-data/ts-0/wal/instance
I20260925 22:35:54.494318 7131 external_mini_cluster.cc:1428] Setting key 3a594967479c528dcdbd9e57a6604acb
Sep 25 22:35:54 dist-test-slave-wjdr krb5kdc[7138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375754, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.193@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
I20260925 22:35:54.499217 7221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:35:54.486354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.193:50215 (local address 127.6.246.254:46343)
0925 22:35:54.486495 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 22:35:54.486499 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:35:54.487364 (+ 865us) server_negotiation.cc:407] Connection header received
0925 22:35:54.489047 (+ 1683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:35:54.489050 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:35:54.489110 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:35:54.489199 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:35:54.490393 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:35:54.490944 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:35:54.492098 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:35:54.492348 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:35:54.492464 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:35:54.495111 (+ 2647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:35:54.495119 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:35:54.495121 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:35:54.495145 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:35:54.496835 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:35:54.497492 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:35:54.497498 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:35:54.497501 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:35:54.497611 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:35:54.497957 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:35:54.497963 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:35:54.497965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:35:54.498206 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:35:54.498317 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 22:35:54.498821 (+ 504us) server_negotiation.cc:299] Negotiation successful
0925 22:35:54.498962 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 22:35:54.499799 7376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:35:54.486706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:46343 (local address 127.6.246.193:50215)
0925 22:35:54.487188 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0925 22:35:54.487247 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 22:35:54.488824 (+ 1577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:35:54.489389 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:35:54.489396 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:35:54.489759 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:35:54.490234 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:35:54.490248 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:35:54.491075 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:35:54.491080 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:35:54.491968 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:35:54.491977 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:35:54.492536 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:35:54.492541 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:35:54.492699 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:35:54.493231 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:35:54.493248 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:35:54.494907 (+ 1659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:35:54.496947 (+ 2040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:35:54.496956 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:35:54.496974 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:35:54.497345 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:35:54.497720 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:35:54.497723 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:35:54.497724 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:35:54.497863 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:35:54.498314 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:35:54.498319 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:35:54.498539 (+ 220us) client_negotiation.cc:769] Sending connection context
0925 22:35:54.498688 (+ 149us) client_negotiation.cc:240] Negotiation successful
0925 22:35:54.498958 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":138,"threads_started":1}
I20260925 22:35:54.501044 7374 heartbeater.cc:347] Connected to a master server at 127.6.246.254:46343
I20260925 22:35:54.501286 7374 heartbeater.cc:464] Registering TS with master...
I20260925 22:35:54.501750 7374 heartbeater.cc:511] Master 127.6.246.254:46343 requested a full tablet report, sending...
I20260925 22:35:54.503185 7183 ts_manager.cc:194] Registered new tserver with Master: a2bf5aa00abb4f14b69c5520a25247ae (127.6.246.193:39995)
WARNING: no policy specified for kudu/127.6.246.194@KRBTEST.COM; defaulting to no policy
I20260925 22:35:54.504331 7183 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.246.193@KRBTEST.COM'} at 127.6.246.193:50215
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.194.keytab.
Entry for principal kudu/127.6.246.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.194.keytab.
I20260925 22:35:54.523458 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.194:0
--local_ip_for_outbound_sockets=127.6.246.194
--webserver_interface=127.6.246.194
--webserver_port=0
--tserver_master_addrs=127.6.246.254:46343
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.194.keytab
--principal=kudu/127.6.246.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:35341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:35:54.634919 7379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:35:54.635176 7379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:35:54.635223 7379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:35:54.639288 7379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:35:54.639345 7379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:35:54.639406 7379 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.194
I20260925 22:35:54.644716 7379 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:35341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.194.keytab
--principal=kudu/127.6.246.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.246.194
--webserver_port=0
--tserver_master_addrs=127.6.246.254:46343
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7379
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.194
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:35:54.645671 7379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:35:54.646625 7379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:35:54.656824 7385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:35:54.656883 7384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:35:54.656819 7387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:35:54.657243 7379 server_base.cc:1034] running on GCE node
I20260925 22:35:54.657608 7379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:35:54.658238 7379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:35:54.659426 7379 hybrid_clock.cc:648] HybridClock initialized: now 1790375754659382 us; error 61 us; skew 500 ppm
Sep 25 22:35:54 dist-test-slave-wjdr krb5kdc[7138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375754, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:35:54.662321 7379 init.cc:377] Logged in from keytab as kudu/127.6.246.194@KRBTEST.COM (short username kudu)
I20260925 22:35:54.663203 7379 webserver.cc:460] Webserver started at http://127.6.246.194:40015/ using document root <none> and password file <none>
I20260925 22:35:54.663764 7379 fs_manager.cc:359] Metadata directory not provided
I20260925 22:35:54.663854 7379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:35:54.664077 7379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:35:54.666160 7379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/minicluster-data/ts-1/data/instance:
uuid: "386d442140394d01baa968ef80867606"
format_stamp: "Formatted at 2026-09-25 22:35:54 on dist-test-slave-wjdr"
server_key: "1a6725333b76409574614e4ca032290b"
server_key_iv: "904adc2770bab53047c14b3a69b36253"
server_key_version: "encryptionkey@0"
I20260925 22:35:54.666720 7379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/minicluster-data/ts-1/wal/instance:
uuid: "386d442140394d01baa968ef80867606"
format_stamp: "Formatted at 2026-09-25 22:35:54 on dist-test-slave-wjdr"
server_key: "1a6725333b76409574614e4ca032290b"
server_key_iv: "904adc2770bab53047c14b3a69b36253"
server_key_version: "encryptionkey@0"
I20260925 22:35:54.670795 7379 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260925 22:35:54.673365 7394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:35:54.674552 7379 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260925 22:35:54.674683 7379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/minicluster-data/ts-1/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/minicluster-data/ts-1/wal
uuid: "386d442140394d01baa968ef80867606"
format_stamp: "Formatted at 2026-09-25 22:35:54 on dist-test-slave-wjdr"
server_key: "1a6725333b76409574614e4ca032290b"
server_key_iv: "904adc2770bab53047c14b3a69b36253"
server_key_version: "encryptionkey@0"
I20260925 22:35:54.674793 7379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:35:54.686148 7379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:35:54.688866 7379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:35:54.689060 7379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:35:54.689602 7379 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:35:54.690404 7379 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:35:54.690474 7379 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:35:54.690546 7379 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:35:54.690588 7379 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:35:54.700042 7379 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.194:40355
I20260925 22:35:54.700081 7507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.194:40355 every 8 connection(s)
I20260925 22:35:54.700944 7379 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
I20260925 22:35:54.709546 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 7379
I20260925 22:35:54.709690 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/minicluster-data/ts-1/wal/instance
I20260925 22:35:54.709993 7131 external_mini_cluster.cc:1428] Setting key 304d0f19115c6abf5e4b64668a180321
Sep 25 22:35:54 dist-test-slave-wjdr krb5kdc[7138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375754, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.194@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
I20260925 22:35:54.716056 7221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:35:54.702854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.194:51509 (local address 127.6.246.254:46343)
0925 22:35:54.702997 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 22:35:54.703001 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:35:54.703832 (+ 831us) server_negotiation.cc:407] Connection header received
0925 22:35:54.705275 (+ 1443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:35:54.705279 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:35:54.705335 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:35:54.705441 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:35:54.706587 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:35:54.707172 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:35:54.708328 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:35:54.708618 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:35:54.708727 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:35:54.711648 (+ 2921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:35:54.711654 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:35:54.711657 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:35:54.711681 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:35:54.713599 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:35:54.714265 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:35:54.714268 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:35:54.714269 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:35:54.714357 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:35:54.714741 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:35:54.714746 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:35:54.714751 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:35:54.714991 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:35:54.715097 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 22:35:54.715664 (+ 567us) server_negotiation.cc:299] Negotiation successful
0925 22:35:54.715808 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 22:35:54.716948 7510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:35:54.703208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:46343 (local address 127.6.246.194:51509)
0925 22:35:54.703674 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0925 22:35:54.703715 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 22:35:54.705055 (+ 1340us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:35:54.705607 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:35:54.705613 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:35:54.705957 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:35:54.706444 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:35:54.706456 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:35:54.707317 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:35:54.707321 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:35:54.708193 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:35:54.708202 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:35:54.708775 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:35:54.708779 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:35:54.708945 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:35:54.709557 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:35:54.709574 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:35:54.711468 (+ 1894us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:35:54.713715 (+ 2247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:35:54.713737 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:35:54.713787 (+ 50us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:35:54.714162 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:35:54.714453 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:35:54.714458 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:35:54.714460 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:35:54.714638 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:35:54.715101 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:35:54.715106 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:35:54.715400 (+ 294us) client_negotiation.cc:769] Sending connection context
0925 22:35:54.715559 (+ 159us) client_negotiation.cc:240] Negotiation successful
0925 22:35:54.715945 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":129,"threads_started":1}
I20260925 22:35:54.718508 7508 heartbeater.cc:347] Connected to a master server at 127.6.246.254:46343
I20260925 22:35:54.718770 7508 heartbeater.cc:464] Registering TS with master...
I20260925 22:35:54.719375 7508 heartbeater.cc:511] Master 127.6.246.254:46343 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.6.246.195@KRBTEST.COM; defaulting to no policy
I20260925 22:35:54.720551 7183 ts_manager.cc:194] Registered new tserver with Master: 386d442140394d01baa968ef80867606 (127.6.246.194:40355)
I20260925 22:35:54.721122 7183 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.246.194@KRBTEST.COM'} at 127.6.246.194:51509
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.195.keytab.
Entry for principal kudu/127.6.246.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.195.keytab.
I20260925 22:35:54.739356 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.195:0
--local_ip_for_outbound_sockets=127.6.246.195
--webserver_interface=127.6.246.195
--webserver_port=0
--tserver_master_addrs=127.6.246.254:46343
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.195.keytab
--principal=kudu/127.6.246.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:35341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:35:54.850703 7513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:35:54.850968 7513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:35:54.851011 7513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:35:54.855376 7513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:35:54.855435 7513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:35:54.855527 7513 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.195
I20260925 22:35:54.860973 7513 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:35341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.195.keytab
--principal=kudu/127.6.246.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.246.195
--webserver_port=0
--tserver_master_addrs=127.6.246.254:46343
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7513
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.195
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:35:54.861995 7513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:35:54.862989 7513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:35:54.873124 7518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:35:54.873117 7521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:35:54.873277 7513 server_base.cc:1034] running on GCE node
W20260925 22:35:54.873112 7519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:35:54.873741 7513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:35:54.874437 7513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:35:54.875630 7513 hybrid_clock.cc:648] HybridClock initialized: now 1790375754875614 us; error 41 us; skew 500 ppm
Sep 25 22:35:54 dist-test-slave-wjdr krb5kdc[7138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375754, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:35:54.878345 7513 init.cc:377] Logged in from keytab as kudu/127.6.246.195@KRBTEST.COM (short username kudu)
I20260925 22:35:54.879261 7513 webserver.cc:460] Webserver started at http://127.6.246.195:32857/ using document root <none> and password file <none>
I20260925 22:35:54.879812 7513 fs_manager.cc:359] Metadata directory not provided
I20260925 22:35:54.879899 7513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:35:54.880123 7513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:35:54.882200 7513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/minicluster-data/ts-2/data/instance:
uuid: "11ab53848097409297b760554457aee6"
format_stamp: "Formatted at 2026-09-25 22:35:54 on dist-test-slave-wjdr"
server_key: "158c598cb7ff43affd9d1476ad3a9640"
server_key_iv: "5b614f0890a0c5c55f07f0fec1c42046"
server_key_version: "encryptionkey@0"
I20260925 22:35:54.882754 7513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/minicluster-data/ts-2/wal/instance:
uuid: "11ab53848097409297b760554457aee6"
format_stamp: "Formatted at 2026-09-25 22:35:54 on dist-test-slave-wjdr"
server_key: "158c598cb7ff43affd9d1476ad3a9640"
server_key_iv: "5b614f0890a0c5c55f07f0fec1c42046"
server_key_version: "encryptionkey@0"
I20260925 22:35:54.886436 7513 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 22:35:54.888725 7528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:35:54.889878 7513 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 22:35:54.890005 7513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/minicluster-data/ts-2/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/minicluster-data/ts-2/wal
uuid: "11ab53848097409297b760554457aee6"
format_stamp: "Formatted at 2026-09-25 22:35:54 on dist-test-slave-wjdr"
server_key: "158c598cb7ff43affd9d1476ad3a9640"
server_key_iv: "5b614f0890a0c5c55f07f0fec1c42046"
server_key_version: "encryptionkey@0"
I20260925 22:35:54.890110 7513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:35:54.903110 7513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:35:54.905974 7513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:35:54.906172 7513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:35:54.906723 7513 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:35:54.907500 7513 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:35:54.907569 7513 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:35:54.907639 7513 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:35:54.907682 7513 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:35:54.917611 7513 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.195:38477
I20260925 22:35:54.917634 7641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.195:38477 every 8 connection(s)
I20260925 22:35:54.918567 7513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
I20260925 22:35:54.925625 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 7513
I20260925 22:35:54.925726 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790375752797504-7131-0/minicluster-data/ts-2/wal/instance
I20260925 22:35:54.926010 7131 external_mini_cluster.cc:1428] Setting key 3fa673a69dd56985d7b73e5c8710bc6a
Sep 25 22:35:54 dist-test-slave-wjdr krb5kdc[7138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375754, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.195@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
I20260925 22:35:54.932154 7221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:35:54.920301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.195:34615 (local address 127.6.246.254:46343)
0925 22:35:54.920438 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 22:35:54.920441 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:35:54.921305 (+ 864us) server_negotiation.cc:407] Connection header received
0925 22:35:54.922685 (+ 1380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:35:54.922688 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:35:54.922747 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:35:54.922822 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:35:54.923987 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:35:54.924586 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:35:54.925802 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:35:54.926016 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:35:54.926138 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:35:54.929046 (+ 2908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:35:54.929049 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:35:54.929051 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:35:54.929075 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:35:54.930303 (+ 1228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:35:54.930794 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:35:54.930797 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:35:54.930798 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:35:54.930879 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:35:54.931178 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:35:54.931180 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:35:54.931181 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:35:54.931363 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:35:54.931439 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 22:35:54.931937 (+ 498us) server_negotiation.cc:299] Negotiation successful
0925 22:35:54.932039 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 22:35:54.933023 7644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:35:54.920677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:46343 (local address 127.6.246.195:34615)
0925 22:35:54.921149 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0925 22:35:54.921191 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 22:35:54.922467 (+ 1276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:35:54.922991 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:35:54.922998 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:35:54.923352 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:35:54.923831 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:35:54.923845 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:35:54.924708 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:35:54.924711 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:35:54.925618 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:35:54.925627 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:35:54.926177 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:35:54.926181 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:35:54.926359 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:35:54.927097 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:35:54.927127 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:35:54.928865 (+ 1738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:35:54.930408 (+ 1543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:35:54.930412 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:35:54.930425 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:35:54.930676 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:35:54.930969 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:35:54.930971 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:35:54.930973 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:35:54.931099 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:35:54.931448 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:35:54.931452 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:35:54.931670 (+ 218us) client_negotiation.cc:769] Sending connection context
0925 22:35:54.931822 (+ 152us) client_negotiation.cc:240] Negotiation successful
0925 22:35:54.932091 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":146,"threads_started":1}
I20260925 22:35:54.934152 7642 heartbeater.cc:347] Connected to a master server at 127.6.246.254:46343
I20260925 22:35:54.934362 7642 heartbeater.cc:464] Registering TS with master...
I20260925 22:35:54.934791 7642 heartbeater.cc:511] Master 127.6.246.254:46343 requested a full tablet report, sending...
I20260925 22:35:54.935685 7183 ts_manager.cc:194] Registered new tserver with Master: 11ab53848097409297b760554457aee6 (127.6.246.195:38477)
I20260925 22:35:54.936232 7183 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.246.195@KRBTEST.COM'} at 127.6.246.195:34615
I20260925 22:35:54.940665 7131 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 22:35:54 dist-test-slave-wjdr krb5kdc[7138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375754, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 22:35:54 dist-test-slave-wjdr krb5kdc[7138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375754, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.246.193@KRBTEST.COM
I20260925 22:35:54.967659 7649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:35:54.957110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46178 (local address 127.6.246.193:39995)
0925 22:35:54.957398 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0925 22:35:54.957416 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 22:35:54.957505 (+ 89us) server_negotiation.cc:407] Connection header received
0925 22:35:54.957693 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:35:54.957699 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:35:54.957896 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:35:54.958046 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:35:54.958721 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:35:54.959609 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:35:54.960446 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:35:54.960859 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:35:54.960980 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:35:54.963736 (+ 2756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:35:54.963746 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:35:54.963768 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:35:54.963792 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:35:54.965442 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:35:54.965891 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:35:54.965898 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:35:54.965902 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:35:54.965988 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:35:54.966283 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:35:54.966289 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:35:54.966291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:35:54.966539 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:35:54.966765 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0925 22:35:54.966938 (+ 173us) server_negotiation.cc:299] Negotiation successful
0925 22:35:54.967093 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":90,"threads_started":1}
W20260925 22:35:54.968822 7288 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46178
Sep 25 22:35:54 dist-test-slave-wjdr krb5kdc[7138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 22:35:54 dist-test-slave-wjdr krb5kdc[7138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.246.193@KRBTEST.COM
I20260925 22:35:54.995311 7649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:35:54.986778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46194 (local address 127.6.246.193:39995)
0925 22:35:54.986929 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 22:35:54.986933 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:35:54.986948 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:35:54.987067 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:35:54.987070 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:35:54.987117 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:35:54.987194 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:35:54.987947 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:35:54.988538 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:35:54.989404 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:35:54.989642 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:35:54.989722 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:35:54.992341 (+ 2619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:35:54.992345 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:35:54.992347 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:35:54.992372 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:35:54.993811 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:35:54.994197 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:35:54.994200 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:35:54.994201 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:35:54.994286 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:35:54.994538 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:35:54.994540 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:35:54.994541 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:35:54.994754 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:35:54.994883 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 22:35:54.995070 (+ 187us) server_negotiation.cc:299] Negotiation successful
0925 22:35:54.995171 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 22:35:54.996414 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 7245
I20260925 22:35:55.002604 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 7379
I20260925 22:35:55.009889 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 7513
I20260925 22:35:55.015460 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 7151
2026-09-25T22:35:55Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2212 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:35:55 dist-test-slave-wjdr krb5kdc[7656](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:35:55 dist-test-slave-wjdr krb5kdc[7656](info): set up 2 sockets
Sep 25 22:35:55 dist-test-slave-wjdr krb5kdc[7656](info): commencing operation
krb5kdc: starting...
W20260925 22:35:56.132413 7131 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Sep 25 22:35:56 dist-test-slave-wjdr krb5kdc[7656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:35:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:35:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.246.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab.
Entry for principal kudu/127.6.246.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab.
I20260925 22:35:56.255374 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:33869
--webserver_interface=127.6.246.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab
--principal=kudu/127.6.246.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:40959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.246.254:33869
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:35:56.363348 7669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:35:56.363652 7669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:35:56.363730 7669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:35:56.367659 7669 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:35:56.367753 7669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:35:56.367789 7669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:35:56.367841 7669 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:35:56.367878 7669 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:35:56.373037 7669 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:40959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.246.254:33869
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab
--principal=kudu/127.6.246.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:33869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.246.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7669
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:35:56.374116 7669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:35:56.375128 7669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:35:56.383803 7675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:35:56.383803 7674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:35:56.383934 7669 server_base.cc:1034] running on GCE node
W20260925 22:35:56.384109 7677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:35:56.384737 7669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:35:56.385839 7669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:35:56.387037 7669 hybrid_clock.cc:648] HybridClock initialized: now 1790375756387018 us; error 31 us; skew 500 ppm
Sep 25 22:35:56 dist-test-slave-wjdr krb5kdc[7656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375756, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:35:56.390029 7669 init.cc:377] Logged in from keytab as kudu/127.6.246.254@KRBTEST.COM (short username kudu)
I20260925 22:35:56.390878 7669 webserver.cc:460] Webserver started at http://127.6.246.254:40057/ using document root <none> and password file <none>
I20260925 22:35:56.391402 7669 fs_manager.cc:359] Metadata directory not provided
I20260925 22:35:56.391460 7669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:35:56.391624 7669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:35:56.393752 7669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/minicluster-data/master-0/data/instance:
uuid: "3f90298fc738448cb2845ad8cd6db2a7"
format_stamp: "Formatted at 2026-09-25 22:35:56 on dist-test-slave-wjdr"
server_key: "232b45fc3b8c151abd94cd3a92d75197"
server_key_iv: "004bcfb4bce3277cf9b2ca91dd7426de"
server_key_version: "encryptionkey@0"
I20260925 22:35:56.394274 7669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/minicluster-data/master-0/wal/instance:
uuid: "3f90298fc738448cb2845ad8cd6db2a7"
format_stamp: "Formatted at 2026-09-25 22:35:56 on dist-test-slave-wjdr"
server_key: "232b45fc3b8c151abd94cd3a92d75197"
server_key_iv: "004bcfb4bce3277cf9b2ca91dd7426de"
server_key_version: "encryptionkey@0"
I20260925 22:35:56.397912 7669 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:35:56.400202 7684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:35:56.401522 7669 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:35:56.401623 7669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/minicluster-data/master-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/minicluster-data/master-0/wal
uuid: "3f90298fc738448cb2845ad8cd6db2a7"
format_stamp: "Formatted at 2026-09-25 22:35:56 on dist-test-slave-wjdr"
server_key: "232b45fc3b8c151abd94cd3a92d75197"
server_key_iv: "004bcfb4bce3277cf9b2ca91dd7426de"
server_key_version: "encryptionkey@0"
I20260925 22:35:56.401707 7669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:35:56.414219 7669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:35:56.417086 7669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:35:56.417233 7669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:35:56.424203 7669 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.254:33869
I20260925 22:35:56.424269 7736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.254:33869 every 8 connection(s)
I20260925 22:35:56.425233 7669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
I20260925 22:35:56.428009 7737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:35:56.431254 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 7669
I20260925 22:35:56.431349 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/minicluster-data/master-0/wal/instance
I20260925 22:35:56.431634 7131 external_mini_cluster.cc:1428] Setting key 09016fd611a63f3097bee710b8fd7bbd
I20260925 22:35:56.433998 7737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f90298fc738448cb2845ad8cd6db2a7: Bootstrap starting.
I20260925 22:35:56.436673 7737 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f90298fc738448cb2845ad8cd6db2a7: Neither blocks nor log segments found. Creating new log.
I20260925 22:35:56.437577 7737 log.cc:824] T 00000000000000000000000000000000 P 3f90298fc738448cb2845ad8cd6db2a7: Log is configured to *not* fsync() on all Append() calls
I20260925 22:35:56.439817 7737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f90298fc738448cb2845ad8cd6db2a7: No bootstrap required, opened a new log
Sep 25 22:35:56 dist-test-slave-wjdr krb5kdc[7656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
I20260925 22:35:56.443262 7737 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f90298fc738448cb2845ad8cd6db2a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f90298fc738448cb2845ad8cd6db2a7" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 33869 } }
I20260925 22:35:56.443557 7737 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f90298fc738448cb2845ad8cd6db2a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:35:56.443639 7737 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f90298fc738448cb2845ad8cd6db2a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f90298fc738448cb2845ad8cd6db2a7, State: Initialized, Role: FOLLOWER
I20260925 22:35:56.444151 7737 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f90298fc738448cb2845ad8cd6db2a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f90298fc738448cb2845ad8cd6db2a7" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 33869 } }
I20260925 22:35:56.444279 7737 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f90298fc738448cb2845ad8cd6db2a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:35:56.444360 7737 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f90298fc738448cb2845ad8cd6db2a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:35:56.444489 7737 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f90298fc738448cb2845ad8cd6db2a7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:35:56.445371 7737 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f90298fc738448cb2845ad8cd6db2a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f90298fc738448cb2845ad8cd6db2a7" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 33869 } }
I20260925 22:35:56.445715 7737 leader_election.cc:302] T 00000000000000000000000000000000 P 3f90298fc738448cb2845ad8cd6db2a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f90298fc738448cb2845ad8cd6db2a7; no voters:
I20260925 22:35:56.446049 7737 leader_election.cc:288] T 00000000000000000000000000000000 P 3f90298fc738448cb2845ad8cd6db2a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:35:56.446117 7742 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f90298fc738448cb2845ad8cd6db2a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:35:56.446350 7742 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f90298fc738448cb2845ad8cd6db2a7 [term 1 LEADER]: Becoming Leader. State: Replica: 3f90298fc738448cb2845ad8cd6db2a7, State: Running, Role: LEADER
I20260925 22:35:56.446694 7742 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f90298fc738448cb2845ad8cd6db2a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f90298fc738448cb2845ad8cd6db2a7" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 33869 } }
I20260925 22:35:56.447332 7737 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f90298fc738448cb2845ad8cd6db2a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:35:56.447762 7740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:35:56.433107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54490 (local address 127.6.246.254:33869)
0925 22:35:56.433540 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0925 22:35:56.433555 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:35:56.433593 (+ 38us) server_negotiation.cc:407] Connection header received
0925 22:35:56.434273 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:35:56.434291 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:35:56.434627 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:35:56.434904 (+ 277us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:35:56.435816 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:35:56.436927 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:35:56.438102 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:35:56.438675 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:35:56.438796 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:35:56.441594 (+ 2798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:35:56.441599 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:35:56.441612 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:35:56.441639 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:35:56.444373 (+ 2734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:35:56.444892 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:35:56.444898 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:35:56.444904 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:35:56.445024 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:35:56.445305 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:35:56.445310 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:35:56.445311 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:35:56.445781 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:35:56.446025 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0925 22:35:56.446630 (+ 605us) server_negotiation.cc:299] Negotiation successful
0925 22:35:56.446948 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":147,"threads_started":1}
I20260925 22:35:56.447934 7742 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f90298fc738448cb2845ad8cd6db2a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f90298fc738448cb2845ad8cd6db2a7. Latest consensus state: current_term: 1 leader_uuid: "3f90298fc738448cb2845ad8cd6db2a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f90298fc738448cb2845ad8cd6db2a7" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 33869 } } }
I20260925 22:35:56.448096 7742 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f90298fc738448cb2845ad8cd6db2a7 [sys.catalog]: This master's current role is: LEADER
I20260925 22:35:56.448675 7747 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:35:56.448669 7743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f90298fc738448cb2845ad8cd6db2a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f90298fc738448cb2845ad8cd6db2a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f90298fc738448cb2845ad8cd6db2a7" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 33869 } } }
I20260925 22:35:56.448767 7743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f90298fc738448cb2845ad8cd6db2a7 [sys.catalog]: This master's current role is: LEADER
I20260925 22:35:56.451900 7747 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:35:56.457414 7747 catalog_manager.cc:1347] Generated new cluster ID: 47d9356961f34d26acdb0c7d1045d15a
I20260925 22:35:56.457480 7747 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:35:56.484185 7747 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:35:56.485230 7747 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:35:56.490569 7747 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f90298fc738448cb2845ad8cd6db2a7: Generated new TSK 0
I20260925 22:35:56.491395 7747 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.246.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.193.keytab.
Entry for principal kudu/127.6.246.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.193.keytab.
I20260925 22:35:56.529347 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.193:0
--local_ip_for_outbound_sockets=127.6.246.193
--webserver_interface=127.6.246.193
--webserver_port=0
--tserver_master_addrs=127.6.246.254:33869
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.193.keytab
--principal=kudu/127.6.246.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:40959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:35:56.659108 7763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:35:56.659400 7763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:35:56.659471 7763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:35:56.663568 7763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:35:56.663664 7763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:35:56.663754 7763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.193
I20260925 22:35:56.669133 7763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:40959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.193.keytab
--principal=kudu/127.6.246.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.246.193
--webserver_port=0
--tserver_master_addrs=127.6.246.254:33869
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.193
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:35:56.670178 7763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:35:56.671254 7763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:35:56.680636 7769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:35:56.680639 7768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:35:56.680698 7771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:35:56.680804 7763 server_base.cc:1034] running on GCE node
I20260925 22:35:56.681226 7763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:35:56.681804 7763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:35:56.682982 7763 hybrid_clock.cc:648] HybridClock initialized: now 1790375756682947 us; error 49 us; skew 500 ppm
Sep 25 22:35:56 dist-test-slave-wjdr krb5kdc[7656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375756, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:35:56.685752 7763 init.cc:377] Logged in from keytab as kudu/127.6.246.193@KRBTEST.COM (short username kudu)
I20260925 22:35:56.686617 7763 webserver.cc:460] Webserver started at http://127.6.246.193:39033/ using document root <none> and password file <none>
I20260925 22:35:56.687165 7763 fs_manager.cc:359] Metadata directory not provided
I20260925 22:35:56.687266 7763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:35:56.687485 7763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:35:56.689514 7763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/minicluster-data/ts-0/data/instance:
uuid: "66d33d841093462b8ffe26045545d6e6"
format_stamp: "Formatted at 2026-09-25 22:35:56 on dist-test-slave-wjdr"
server_key: "8e565e8acdb7577761750af3e6740a19"
server_key_iv: "68f52db1c3aea6e2db2953dc98eabc7a"
server_key_version: "encryptionkey@0"
I20260925 22:35:56.690060 7763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/minicluster-data/ts-0/wal/instance:
uuid: "66d33d841093462b8ffe26045545d6e6"
format_stamp: "Formatted at 2026-09-25 22:35:56 on dist-test-slave-wjdr"
server_key: "8e565e8acdb7577761750af3e6740a19"
server_key_iv: "68f52db1c3aea6e2db2953dc98eabc7a"
server_key_version: "encryptionkey@0"
I20260925 22:35:56.693614 7763 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260925 22:35:56.696030 7778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:35:56.697278 7763 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:35:56.697435 7763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/minicluster-data/ts-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/minicluster-data/ts-0/wal
uuid: "66d33d841093462b8ffe26045545d6e6"
format_stamp: "Formatted at 2026-09-25 22:35:56 on dist-test-slave-wjdr"
server_key: "8e565e8acdb7577761750af3e6740a19"
server_key_iv: "68f52db1c3aea6e2db2953dc98eabc7a"
server_key_version: "encryptionkey@0"
I20260925 22:35:56.697539 7763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:35:56.707891 7763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:35:56.710906 7763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:35:56.711089 7763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:35:56.711632 7763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:35:56.712430 7763 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:35:56.712563 7763 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:35:56.712631 7763 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:35:56.712674 7763 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:35:56.721626 7763 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.193:46147
I20260925 22:35:56.721657 7891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.193:46147 every 8 connection(s)
I20260925 22:35:56.722435 7763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
I20260925 22:35:56.725855 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 7763
I20260925 22:35:56.725993 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/minicluster-data/ts-0/wal/instance
I20260925 22:35:56.726313 7131 external_mini_cluster.cc:1428] Setting key a47c74a0e79d7d5d4b5f20d9cc5e2033
Sep 25 22:35:56 dist-test-slave-wjdr krb5kdc[7656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375756, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.193@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.246.194@KRBTEST.COM; defaulting to no policy
I20260925 22:35:56.737183 7740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:35:56.724279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.193:42391 (local address 127.6.246.254:33869)
0925 22:35:56.724414 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 22:35:56.724418 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:35:56.725312 (+ 894us) server_negotiation.cc:407] Connection header received
0925 22:35:56.727034 (+ 1722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:35:56.727037 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:35:56.727090 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:35:56.727197 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:35:56.728594 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:35:56.729151 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:35:56.730278 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:35:56.730529 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:35:56.730636 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:35:56.733418 (+ 2782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:35:56.733421 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:35:56.733423 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:35:56.733445 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:35:56.734852 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:35:56.735656 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:35:56.735658 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:35:56.735659 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:35:56.735727 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:35:56.736180 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:35:56.736183 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:35:56.736184 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:35:56.736354 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:35:56.736426 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 22:35:56.736970 (+ 544us) server_negotiation.cc:299] Negotiation successful
0925 22:35:56.737069 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 22:35:56.737891 7894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:35:56.724700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:33869 (local address 127.6.246.193:42391)
0925 22:35:56.725152 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0925 22:35:56.725192 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 22:35:56.726836 (+ 1644us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:35:56.727360 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:35:56.727366 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:35:56.727769 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:35:56.728419 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:35:56.728441 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:35:56.729272 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:35:56.729276 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:35:56.730155 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:35:56.730165 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:35:56.730673 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:35:56.730677 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:35:56.730841 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:35:56.731435 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:35:56.731450 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:35:56.733245 (+ 1795us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:35:56.735015 (+ 1770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:35:56.735024 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:35:56.735043 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:35:56.735551 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:35:56.735882 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:35:56.735885 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:35:56.735887 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:35:56.736083 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:35:56.736509 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:35:56.736514 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:35:56.736754 (+ 240us) client_negotiation.cc:769] Sending connection context
0925 22:35:56.736897 (+ 143us) client_negotiation.cc:240] Negotiation successful
0925 22:35:56.737121 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":122,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.194@KRBTEST.COM" created.
I20260925 22:35:56.739145 7892 heartbeater.cc:347] Connected to a master server at 127.6.246.254:33869
I20260925 22:35:56.739372 7892 heartbeater.cc:464] Registering TS with master...
I20260925 22:35:56.739830 7892 heartbeater.cc:511] Master 127.6.246.254:33869 requested a full tablet report, sending...
I20260925 22:35:56.741482 7701 ts_manager.cc:194] Registered new tserver with Master: 66d33d841093462b8ffe26045545d6e6 (127.6.246.193:46147)
I20260925 22:35:56.743113 7701 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.246.193@KRBTEST.COM'} at 127.6.246.193:42391
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.194.keytab.
Entry for principal kudu/127.6.246.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.194.keytab.
I20260925 22:35:56.755581 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.194:0
--local_ip_for_outbound_sockets=127.6.246.194
--webserver_interface=127.6.246.194
--webserver_port=0
--tserver_master_addrs=127.6.246.254:33869
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.194.keytab
--principal=kudu/127.6.246.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:40959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:35:56.863422 7897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:35:56.863668 7897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:35:56.863710 7897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:35:56.867604 7897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:35:56.867656 7897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:35:56.867713 7897 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.194
I20260925 22:35:56.872745 7897 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:40959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.194.keytab
--principal=kudu/127.6.246.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.246.194
--webserver_port=0
--tserver_master_addrs=127.6.246.254:33869
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.194
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:35:56.873786 7897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:35:56.874786 7897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:35:56.884344 7902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:35:56.884344 7905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:35:56.884349 7903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:35:56.884708 7897 server_base.cc:1034] running on GCE node
I20260925 22:35:56.885140 7897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:35:56.885803 7897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:35:56.886991 7897 hybrid_clock.cc:648] HybridClock initialized: now 1790375756886972 us; error 29 us; skew 500 ppm
Sep 25 22:35:56 dist-test-slave-wjdr krb5kdc[7656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375756, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:35:56.889824 7897 init.cc:377] Logged in from keytab as kudu/127.6.246.194@KRBTEST.COM (short username kudu)
I20260925 22:35:56.890720 7897 webserver.cc:460] Webserver started at http://127.6.246.194:35487/ using document root <none> and password file <none>
I20260925 22:35:56.891234 7897 fs_manager.cc:359] Metadata directory not provided
I20260925 22:35:56.891292 7897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:35:56.891449 7897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:35:56.893460 7897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/minicluster-data/ts-1/data/instance:
uuid: "cea6625f0168438d8efdf01b9ccede02"
format_stamp: "Formatted at 2026-09-25 22:35:56 on dist-test-slave-wjdr"
server_key: "b3bcd792878407228dd13a946c088fd4"
server_key_iv: "22728f84cb09ce548d9c5874aeeabc25"
server_key_version: "encryptionkey@0"
I20260925 22:35:56.893997 7897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/minicluster-data/ts-1/wal/instance:
uuid: "cea6625f0168438d8efdf01b9ccede02"
format_stamp: "Formatted at 2026-09-25 22:35:56 on dist-test-slave-wjdr"
server_key: "b3bcd792878407228dd13a946c088fd4"
server_key_iv: "22728f84cb09ce548d9c5874aeeabc25"
server_key_version: "encryptionkey@0"
I20260925 22:35:56.897526 7897 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:35:56.899852 7912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:35:56.901242 7897 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:35:56.901357 7897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/minicluster-data/ts-1/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/minicluster-data/ts-1/wal
uuid: "cea6625f0168438d8efdf01b9ccede02"
format_stamp: "Formatted at 2026-09-25 22:35:56 on dist-test-slave-wjdr"
server_key: "b3bcd792878407228dd13a946c088fd4"
server_key_iv: "22728f84cb09ce548d9c5874aeeabc25"
server_key_version: "encryptionkey@0"
I20260925 22:35:56.901459 7897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:35:56.929267 7897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:35:56.932119 7897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:35:56.932272 7897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:35:56.932909 7897 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:35:56.933689 7897 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:35:56.933768 7897 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:35:56.933812 7897 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:35:56.933828 7897 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:35:56.942759 7897 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.194:39623
I20260925 22:35:56.942780 8025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.194:39623 every 8 connection(s)
I20260925 22:35:56.943670 7897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
I20260925 22:35:56.951959 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 7897
I20260925 22:35:56.952050 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/minicluster-data/ts-1/wal/instance
I20260925 22:35:56.952368 7131 external_mini_cluster.cc:1428] Setting key 9996fdb8adae2d08a7fb10be4622a5fe
Sep 25 22:35:56 dist-test-slave-wjdr krb5kdc[7656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375756, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.194@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
I20260925 22:35:56.958071 7740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:35:56.945587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.194:59865 (local address 127.6.246.254:33869)
0925 22:35:56.945727 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 22:35:56.945730 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:35:56.946577 (+ 847us) server_negotiation.cc:407] Connection header received
0925 22:35:56.948044 (+ 1467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:35:56.948047 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:35:56.948102 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:35:56.948184 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:35:56.949587 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:35:56.950183 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:35:56.951327 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:35:56.951569 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:35:56.951674 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:35:56.954379 (+ 2705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:35:56.954382 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:35:56.954384 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:35:56.954407 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:35:56.955921 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:35:56.956440 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:35:56.956445 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:35:56.956447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:35:56.956565 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:35:56.956905 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:35:56.956909 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:35:56.956911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:35:56.957089 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:35:56.957196 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 22:35:56.957780 (+ 584us) server_negotiation.cc:299] Negotiation successful
0925 22:35:56.957894 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 22:35:56.958843 8028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:35:56.945947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:33869 (local address 127.6.246.194:59865)
0925 22:35:56.946401 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0925 22:35:56.946444 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 22:35:56.947822 (+ 1378us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:35:56.948356 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:35:56.948362 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:35:56.948902 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:35:56.949442 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:35:56.949456 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:35:56.950312 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:35:56.950320 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:35:56.951195 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:35:56.951204 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:35:56.951736 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:35:56.951741 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:35:56.951906 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:35:56.952625 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:35:56.952642 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:35:56.954202 (+ 1560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:35:56.956044 (+ 1842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:35:56.956051 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:35:56.956066 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:35:56.956332 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:35:56.956662 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:35:56.956666 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:35:56.956668 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:35:56.956808 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:35:56.957194 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:35:56.957200 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:35:56.957530 (+ 330us) client_negotiation.cc:769] Sending connection context
0925 22:35:56.957726 (+ 196us) client_negotiation.cc:240] Negotiation successful
0925 22:35:56.958036 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":110,"threads_started":1}
I20260925 22:35:56.960158 8026 heartbeater.cc:347] Connected to a master server at 127.6.246.254:33869
I20260925 22:35:56.960422 8026 heartbeater.cc:464] Registering TS with master...
I20260925 22:35:56.960927 8026 heartbeater.cc:511] Master 127.6.246.254:33869 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.6.246.195@KRBTEST.COM; defaulting to no policy
I20260925 22:35:56.962001 7701 ts_manager.cc:194] Registered new tserver with Master: cea6625f0168438d8efdf01b9ccede02 (127.6.246.194:39623)
I20260925 22:35:56.962569 7701 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.246.194@KRBTEST.COM'} at 127.6.246.194:59865
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.195.keytab.
Entry for principal kudu/127.6.246.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.195.keytab.
I20260925 22:35:56.979624 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.195:0
--local_ip_for_outbound_sockets=127.6.246.195
--webserver_interface=127.6.246.195
--webserver_port=0
--tserver_master_addrs=127.6.246.254:33869
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.195.keytab
--principal=kudu/127.6.246.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:40959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:35:57.088195 8031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:35:57.088462 8031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:35:57.088567 8031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:35:57.092595 8031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:35:57.092664 8031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:35:57.092748 8031 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.195
I20260925 22:35:57.097891 8031 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:40959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.195.keytab
--principal=kudu/127.6.246.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.246.195
--webserver_port=0
--tserver_master_addrs=127.6.246.254:33869
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8031
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.195
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:35:57.098824 8031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:35:57.099805 8031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 22:35:57.109833 8031 server_base.cc:1034] running on GCE node
W20260925 22:35:57.109750 8037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:35:57.109754 8036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:35:57.109754 8039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:35:57.110458 8031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:35:57.111052 8031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:35:57.112236 8031 hybrid_clock.cc:648] HybridClock initialized: now 1790375757112202 us; error 50 us; skew 500 ppm
Sep 25 22:35:57 dist-test-slave-wjdr krb5kdc[7656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375757, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:35:57.118819 8031 init.cc:377] Logged in from keytab as kudu/127.6.246.195@KRBTEST.COM (short username kudu)
I20260925 22:35:57.119743 8031 webserver.cc:460] Webserver started at http://127.6.246.195:37019/ using document root <none> and password file <none>
I20260925 22:35:57.120296 8031 fs_manager.cc:359] Metadata directory not provided
I20260925 22:35:57.120386 8031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:35:57.120646 8031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:35:57.122692 8031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/minicluster-data/ts-2/data/instance:
uuid: "9685da8e324f4409a28b0ecc91fc1d48"
format_stamp: "Formatted at 2026-09-25 22:35:57 on dist-test-slave-wjdr"
server_key: "68cee2f1c771ac17b6cb4bdb1ffc0b47"
server_key_iv: "e045ac03b869895e267f751345036971"
server_key_version: "encryptionkey@0"
I20260925 22:35:57.123245 8031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/minicluster-data/ts-2/wal/instance:
uuid: "9685da8e324f4409a28b0ecc91fc1d48"
format_stamp: "Formatted at 2026-09-25 22:35:57 on dist-test-slave-wjdr"
server_key: "68cee2f1c771ac17b6cb4bdb1ffc0b47"
server_key_iv: "e045ac03b869895e267f751345036971"
server_key_version: "encryptionkey@0"
I20260925 22:35:57.126983 8031 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 22:35:57.129387 8046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:35:57.130527 8031 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:35:57.130671 8031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/minicluster-data/ts-2/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/minicluster-data/ts-2/wal
uuid: "9685da8e324f4409a28b0ecc91fc1d48"
format_stamp: "Formatted at 2026-09-25 22:35:57 on dist-test-slave-wjdr"
server_key: "68cee2f1c771ac17b6cb4bdb1ffc0b47"
server_key_iv: "e045ac03b869895e267f751345036971"
server_key_version: "encryptionkey@0"
I20260925 22:35:57.130771 8031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:35:57.152751 8031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:35:57.155911 8031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:35:57.156106 8031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:35:57.156684 8031 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:35:57.157466 8031 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:35:57.157534 8031 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:35:57.157603 8031 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:35:57.157653 8031 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:35:57.166657 8031 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.195:39097
I20260925 22:35:57.166685 8159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.195:39097 every 8 connection(s)
I20260925 22:35:57.167486 8031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
I20260925 22:35:57.175443 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 8031
I20260925 22:35:57.175571 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790375752797504-7131-0/minicluster-data/ts-2/wal/instance
I20260925 22:35:57.175875 7131 external_mini_cluster.cc:1428] Setting key 42e4c8dbed5b863d9ce161f135d6216d
Sep 25 22:35:57 dist-test-slave-wjdr krb5kdc[7656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375757, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.195@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
I20260925 22:35:57.181227 7740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:35:57.169434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.195:48223 (local address 127.6.246.254:33869)
0925 22:35:57.169673 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0925 22:35:57.169678 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:35:57.170413 (+ 735us) server_negotiation.cc:407] Connection header received
0925 22:35:57.171804 (+ 1391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:35:57.171808 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:35:57.171866 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:35:57.171949 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:35:57.173110 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:35:57.173655 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:35:57.174814 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:35:57.175042 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:35:57.175152 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:35:57.178063 (+ 2911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:35:57.178066 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:35:57.178071 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:35:57.178099 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:35:57.179413 (+ 1314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:35:57.179854 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:35:57.179856 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:35:57.179857 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:35:57.179926 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:35:57.180202 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:35:57.180204 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:35:57.180205 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:35:57.180375 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:35:57.180518 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0925 22:35:57.181021 (+ 503us) server_negotiation.cc:299] Negotiation successful
0925 22:35:57.181115 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136}
I20260925 22:35:57.181969 8162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:35:57.169771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:33869 (local address 127.6.246.195:48223)
0925 22:35:57.170239 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0925 22:35:57.170297 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 22:35:57.171580 (+ 1283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:35:57.172114 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:35:57.172122 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:35:57.172484 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:35:57.172948 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:35:57.172961 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:35:57.173803 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:35:57.173814 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:35:57.174669 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:35:57.174677 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:35:57.175186 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:35:57.175214 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:35:57.175380 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:35:57.176045 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:35:57.176061 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:35:57.177856 (+ 1795us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:35:57.179511 (+ 1655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:35:57.179515 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:35:57.179526 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:35:57.179769 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:35:57.180001 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:35:57.180003 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:35:57.180005 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:35:57.180127 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:35:57.180511 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:35:57.180518 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:35:57.180787 (+ 269us) client_negotiation.cc:769] Sending connection context
0925 22:35:57.180964 (+ 177us) client_negotiation.cc:240] Negotiation successful
0925 22:35:57.181196 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":111,"threads_started":1}
I20260925 22:35:57.183056 8160 heartbeater.cc:347] Connected to a master server at 127.6.246.254:33869
I20260925 22:35:57.183296 8160 heartbeater.cc:464] Registering TS with master...
I20260925 22:35:57.183812 8160 heartbeater.cc:511] Master 127.6.246.254:33869 requested a full tablet report, sending...
I20260925 22:35:57.184870 7701 ts_manager.cc:194] Registered new tserver with Master: 9685da8e324f4409a28b0ecc91fc1d48 (127.6.246.195:39097)
I20260925 22:35:57.185438 7701 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.246.195@KRBTEST.COM'} at 127.6.246.195:48223
I20260925 22:35:57.189630 7131 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:35:57.199698 7740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:35:57.192869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54500 (local address 127.6.246.254:33869)
0925 22:35:57.192982 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0925 22:35:57.192986 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:35:57.193175 (+ 189us) server_negotiation.cc:407] Connection header received
0925 22:35:57.193279 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:35:57.193281 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:35:57.193315 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:35:57.193385 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:35:57.194046 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:35:57.194597 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:35:57.195352 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:35:57.195557 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:35:57.195635 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:35:57.197007 (+ 1372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:35:57.197010 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:35:57.197012 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:35:57.197032 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:35:57.198218 (+ 1186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:35:57.198595 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:35:57.198597 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:35:57.198599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:35:57.198673 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:35:57.198918 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:35:57.198920 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:35:57.198921 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:35:57.199107 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:35:57.199208 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 22:35:57.199404 (+ 196us) server_negotiation.cc:299] Negotiation successful
0925 22:35:57.199488 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 22:35:57.205554 7701 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54500:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:35:57.210179 7701 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:35:57.228593 8172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:35:57.220681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.195:39097 (local address 127.0.0.1:46282)
0925 22:35:57.221578 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0925 22:35:57.221644 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0925 22:35:57.221772 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:35:57.223838 (+ 2066us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:35:57.223842 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:35:57.223860 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:35:57.223959 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:35:57.223966 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:35:57.225580 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:35:57.225586 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:35:57.226962 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:35:57.226975 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:35:57.228158 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:35:57.228162 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:35:57.228239 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:35:57.228270 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 22:35:57.228320 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 22:35:57.228377 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":712,"mutex_wait_us":52,"thread_start_us":71,"threads_started":1}
I20260925 22:35:57.228901 8174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:35:57.221727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42456 (local address 127.6.246.194:39623)
0925 22:35:57.222407 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0925 22:35:57.222418 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 22:35:57.222439 (+ 21us) server_negotiation.cc:407] Connection header received
0925 22:35:57.222505 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:35:57.222509 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:35:57.222753 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:35:57.222944 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:35:57.223242 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:35:57.224803 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:35:57.226719 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:35:57.227665 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:35:57.227791 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:35:57.227962 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0925 22:35:57.228601 (+ 639us) server_negotiation.cc:299] Negotiation successful
0925 22:35:57.228731 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":147,"threads_started":1}
I20260925 22:35:57.229156 8175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:35:57.221788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46282 (local address 127.6.246.195:39097)
0925 22:35:57.223204 (+ 1416us) server_negotiation.cc:206] Beginning negotiation
0925 22:35:57.223218 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 22:35:57.223241 (+ 23us) server_negotiation.cc:407] Connection header received
0925 22:35:57.223324 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:35:57.223329 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:35:57.223678 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:35:57.223869 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:35:57.224070 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:35:57.225401 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:35:57.227129 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:35:57.228034 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:35:57.228652 (+ 618us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:35:57.228852 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0925 22:35:57.228906 (+ 54us) server_negotiation.cc:299] Negotiation successful
0925 22:35:57.228995 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1290,"thread_start_us":131,"threads_started":1}
I20260925 22:35:57.228394 8173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:35:57.221241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.194:39623 (local address 127.0.0.1:42456)
0925 22:35:57.222034 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0925 22:35:57.222050 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:35:57.222130 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:35:57.222931 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:35:57.222938 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:35:57.222963 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:35:57.223108 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:35:57.223115 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:35:57.224968 (+ 1853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:35:57.224975 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:35:57.226514 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:35:57.226523 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:35:57.227798 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:35:57.227802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:35:57.227914 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:35:57.227956 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 22:35:57.228079 (+ 123us) client_negotiation.cc:240] Negotiation successful
0925 22:35:57.228208 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":728,"mutex_wait_us":1,"thread_start_us":90,"threads_started":1}
I20260925 22:35:57.230261 8171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:35:57.221376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.193:46147 (local address 127.0.0.1:41654)
0925 22:35:57.222798 (+ 1422us) negotiation.cc:107] Waiting for socket to connect
0925 22:35:57.222816 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 22:35:57.222930 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:35:57.223911 (+ 981us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:35:57.223914 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:35:57.223927 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:35:57.224023 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:35:57.224032 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:35:57.226386 (+ 2354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:35:57.226390 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:35:57.228601 (+ 2211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:35:57.228613 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:35:57.229855 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:35:57.229859 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:35:57.229963 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:35:57.229989 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 22:35:57.230045 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 22:35:57.230127 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1328,"mutex_wait_us":26,"thread_start_us":81,"threads_started":1}
I20260925 22:35:57.230916 8176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:35:57.222393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41654 (local address 127.6.246.193:46147)
0925 22:35:57.222720 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0925 22:35:57.222737 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 22:35:57.223269 (+ 532us) server_negotiation.cc:407] Connection header received
0925 22:35:57.223371 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:35:57.223377 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:35:57.223579 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:35:57.223735 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:35:57.224148 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:35:57.226226 (+ 2078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:35:57.228761 (+ 2535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:35:57.229690 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:35:57.230454 (+ 764us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:35:57.230601 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 22:35:57.230652 (+ 51us) server_negotiation.cc:299] Negotiation successful
0925 22:35:57.230746 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":126,"threads_started":1}
I20260925 22:35:57.232197 8094 tablet_service.cc:1467] Processing CreateTablet for tablet 2adf376df4b04dbf915449c89fe638b8 (DEFAULT_TABLE table=test-table [id=69d2d54317824c6e8b6848400db85fbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:35:57.232367 7960 tablet_service.cc:1467] Processing CreateTablet for tablet 2adf376df4b04dbf915449c89fe638b8 (DEFAULT_TABLE table=test-table [id=69d2d54317824c6e8b6848400db85fbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:35:57.233420 8094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2adf376df4b04dbf915449c89fe638b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:35:57.233484 7960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2adf376df4b04dbf915449c89fe638b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:35:57.234439 7826 tablet_service.cc:1467] Processing CreateTablet for tablet 2adf376df4b04dbf915449c89fe638b8 (DEFAULT_TABLE table=test-table [id=69d2d54317824c6e8b6848400db85fbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:35:57.235409 7826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2adf376df4b04dbf915449c89fe638b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:35:57.241509 8177 tablet_bootstrap.cc:492] T 2adf376df4b04dbf915449c89fe638b8 P cea6625f0168438d8efdf01b9ccede02: Bootstrap starting.
I20260925 22:35:57.242975 8178 tablet_bootstrap.cc:492] T 2adf376df4b04dbf915449c89fe638b8 P 9685da8e324f4409a28b0ecc91fc1d48: Bootstrap starting.
I20260925 22:35:57.244689 8177 tablet_bootstrap.cc:654] T 2adf376df4b04dbf915449c89fe638b8 P cea6625f0168438d8efdf01b9ccede02: Neither blocks nor log segments found. Creating new log.
I20260925 22:35:57.245605 8178 tablet_bootstrap.cc:654] T 2adf376df4b04dbf915449c89fe638b8 P 9685da8e324f4409a28b0ecc91fc1d48: Neither blocks nor log segments found. Creating new log.
I20260925 22:35:57.245632 8177 log.cc:824] T 2adf376df4b04dbf915449c89fe638b8 P cea6625f0168438d8efdf01b9ccede02: Log is configured to *not* fsync() on all Append() calls
I20260925 22:35:57.247052 8178 log.cc:824] T 2adf376df4b04dbf915449c89fe638b8 P 9685da8e324f4409a28b0ecc91fc1d48: Log is configured to *not* fsync() on all Append() calls
I20260925 22:35:57.247819 8179 tablet_bootstrap.cc:492] T 2adf376df4b04dbf915449c89fe638b8 P 66d33d841093462b8ffe26045545d6e6: Bootstrap starting.
I20260925 22:35:57.247992 8177 tablet_bootstrap.cc:492] T 2adf376df4b04dbf915449c89fe638b8 P cea6625f0168438d8efdf01b9ccede02: No bootstrap required, opened a new log
I20260925 22:35:57.248202 8177 ts_tablet_manager.cc:1397] T 2adf376df4b04dbf915449c89fe638b8 P cea6625f0168438d8efdf01b9ccede02: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260925 22:35:57.249503 8178 tablet_bootstrap.cc:492] T 2adf376df4b04dbf915449c89fe638b8 P 9685da8e324f4409a28b0ecc91fc1d48: No bootstrap required, opened a new log
I20260925 22:35:57.249715 8178 ts_tablet_manager.cc:1397] T 2adf376df4b04dbf915449c89fe638b8 P 9685da8e324f4409a28b0ecc91fc1d48: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 22:35:57.253338 8178 raft_consensus.cc:348] T 2adf376df4b04dbf915449c89fe638b8 P 9685da8e324f4409a28b0ecc91fc1d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9685da8e324f4409a28b0ecc91fc1d48" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 39097 } } peers { permanent_uuid: "66d33d841093462b8ffe26045545d6e6" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 46147 } } peers { permanent_uuid: "cea6625f0168438d8efdf01b9ccede02" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 39623 } }
I20260925 22:35:57.253610 8178 raft_consensus.cc:374] T 2adf376df4b04dbf915449c89fe638b8 P 9685da8e324f4409a28b0ecc91fc1d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:35:57.253393 8177 raft_consensus.cc:348] T 2adf376df4b04dbf915449c89fe638b8 P cea6625f0168438d8efdf01b9ccede02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9685da8e324f4409a28b0ecc91fc1d48" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 39097 } } peers { permanent_uuid: "66d33d841093462b8ffe26045545d6e6" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 46147 } } peers { permanent_uuid: "cea6625f0168438d8efdf01b9ccede02" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 39623 } }
I20260925 22:35:57.253675 8178 raft_consensus.cc:729] T 2adf376df4b04dbf915449c89fe638b8 P 9685da8e324f4409a28b0ecc91fc1d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9685da8e324f4409a28b0ecc91fc1d48, State: Initialized, Role: FOLLOWER
I20260925 22:35:57.253693 8177 raft_consensus.cc:374] T 2adf376df4b04dbf915449c89fe638b8 P cea6625f0168438d8efdf01b9ccede02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:35:57.253790 8177 raft_consensus.cc:729] T 2adf376df4b04dbf915449c89fe638b8 P cea6625f0168438d8efdf01b9ccede02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cea6625f0168438d8efdf01b9ccede02, State: Initialized, Role: FOLLOWER
I20260925 22:35:57.254190 8178 consensus_queue.cc:260] T 2adf376df4b04dbf915449c89fe638b8 P 9685da8e324f4409a28b0ecc91fc1d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9685da8e324f4409a28b0ecc91fc1d48" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 39097 } } peers { permanent_uuid: "66d33d841093462b8ffe26045545d6e6" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 46147 } } peers { permanent_uuid: "cea6625f0168438d8efdf01b9ccede02" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 39623 } }
I20260925 22:35:57.254271 8177 consensus_queue.cc:260] T 2adf376df4b04dbf915449c89fe638b8 P cea6625f0168438d8efdf01b9ccede02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9685da8e324f4409a28b0ecc91fc1d48" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 39097 } } peers { permanent_uuid: "66d33d841093462b8ffe26045545d6e6" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 46147 } } peers { permanent_uuid: "cea6625f0168438d8efdf01b9ccede02" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 39623 } }
I20260925 22:35:57.254881 8160 heartbeater.cc:503] Master 127.6.246.254:33869 was elected leader, sending a full tablet report...
I20260925 22:35:57.254945 8026 heartbeater.cc:503] Master 127.6.246.254:33869 was elected leader, sending a full tablet report...
I20260925 22:35:57.255373 8177 ts_tablet_manager.cc:1428] T 2adf376df4b04dbf915449c89fe638b8 P cea6625f0168438d8efdf01b9ccede02: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20260925 22:35:57.258244 8179 tablet_bootstrap.cc:654] T 2adf376df4b04dbf915449c89fe638b8 P 66d33d841093462b8ffe26045545d6e6: Neither blocks nor log segments found. Creating new log.
I20260925 22:35:57.259184 8179 log.cc:824] T 2adf376df4b04dbf915449c89fe638b8 P 66d33d841093462b8ffe26045545d6e6: Log is configured to *not* fsync() on all Append() calls
I20260925 22:35:57.260207 8178 ts_tablet_manager.cc:1428] T 2adf376df4b04dbf915449c89fe638b8 P 9685da8e324f4409a28b0ecc91fc1d48: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20260925 22:35:57.261868 8179 tablet_bootstrap.cc:492] T 2adf376df4b04dbf915449c89fe638b8 P 66d33d841093462b8ffe26045545d6e6: No bootstrap required, opened a new log
I20260925 22:35:57.262035 8179 ts_tablet_manager.cc:1397] T 2adf376df4b04dbf915449c89fe638b8 P 66d33d841093462b8ffe26045545d6e6: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20260925 22:35:57.265609 8179 raft_consensus.cc:348] T 2adf376df4b04dbf915449c89fe638b8 P 66d33d841093462b8ffe26045545d6e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9685da8e324f4409a28b0ecc91fc1d48" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 39097 } } peers { permanent_uuid: "66d33d841093462b8ffe26045545d6e6" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 46147 } } peers { permanent_uuid: "cea6625f0168438d8efdf01b9ccede02" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 39623 } }
I20260925 22:35:57.265837 8179 raft_consensus.cc:374] T 2adf376df4b04dbf915449c89fe638b8 P 66d33d841093462b8ffe26045545d6e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:35:57.265873 8179 raft_consensus.cc:729] T 2adf376df4b04dbf915449c89fe638b8 P 66d33d841093462b8ffe26045545d6e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66d33d841093462b8ffe26045545d6e6, State: Initialized, Role: FOLLOWER
I20260925 22:35:57.266233 8179 consensus_queue.cc:260] T 2adf376df4b04dbf915449c89fe638b8 P 66d33d841093462b8ffe26045545d6e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9685da8e324f4409a28b0ecc91fc1d48" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 39097 } } peers { permanent_uuid: "66d33d841093462b8ffe26045545d6e6" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 46147 } } peers { permanent_uuid: "cea6625f0168438d8efdf01b9ccede02" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 39623 } }
I20260925 22:35:57.266812 7892 heartbeater.cc:503] Master 127.6.246.254:33869 was elected leader, sending a full tablet report...
I20260925 22:35:57.267135 8179 ts_tablet_manager.cc:1428] T 2adf376df4b04dbf915449c89fe638b8 P 66d33d841093462b8ffe26045545d6e6: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260925 22:35:57.301399 8182 raft_consensus.cc:482] T 2adf376df4b04dbf915449c89fe638b8 P 9685da8e324f4409a28b0ecc91fc1d48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:35:57.301616 8182 raft_consensus.cc:504] T 2adf376df4b04dbf915449c89fe638b8 P 9685da8e324f4409a28b0ecc91fc1d48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9685da8e324f4409a28b0ecc91fc1d48" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 39097 } } peers { permanent_uuid: "66d33d841093462b8ffe26045545d6e6" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 46147 } } peers { permanent_uuid: "cea6625f0168438d8efdf01b9ccede02" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 39623 } }
I20260925 22:35:57.302692 8182 leader_election.cc:288] T 2adf376df4b04dbf915449c89fe638b8 P 9685da8e324f4409a28b0ecc91fc1d48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66d33d841093462b8ffe26045545d6e6 (127.6.246.193:46147), cea6625f0168438d8efdf01b9ccede02 (127.6.246.194:39623)
I20260925 22:35:57.306428 8162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:35:57.302890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.193:46147 (local address 127.6.246.195:53747)
0925 22:35:57.303041 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 22:35:57.303058 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:35:57.303143 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:35:57.303461 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:35:57.303464 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:35:57.303481 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:35:57.303591 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:35:57.303596 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:35:57.304338 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:35:57.304343 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:35:57.305343 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:35:57.305358 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:35:57.306009 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:35:57.306012 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:35:57.306119 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:35:57.306140 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 22:35:57.306231 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 22:35:57.306301 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260925 22:35:57.306428 8176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:35:57.302978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.195:53747 (local address 127.6.246.193:46147)
0925 22:35:57.303098 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 22:35:57.303102 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:35:57.303119 (+ 17us) server_negotiation.cc:407] Connection header received
0925 22:35:57.303278 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:35:57.303281 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:35:57.303372 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:35:57.303451 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:35:57.303697 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:35:57.304225 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:35:57.305460 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:35:57.305869 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:35:57.306014 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:35:57.306087 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 22:35:57.306253 (+ 166us) server_negotiation.cc:299] Negotiation successful
0925 22:35:57.306319 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 22:35:57.306991 7846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2adf376df4b04dbf915449c89fe638b8" candidate_uuid: "9685da8e324f4409a28b0ecc91fc1d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66d33d841093462b8ffe26045545d6e6" is_pre_election: true
I20260925 22:35:57.307291 7846 raft_consensus.cc:2454] T 2adf376df4b04dbf915449c89fe638b8 P 66d33d841093462b8ffe26045545d6e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9685da8e324f4409a28b0ecc91fc1d48 in term 0.
I20260925 22:35:57.307392 8186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:35:57.302916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.194:39623 (local address 127.6.246.195:60435)
0925 22:35:57.303300 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0925 22:35:57.303315 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:35:57.303440 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:35:57.303689 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:35:57.303692 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:35:57.303707 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:35:57.303831 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:35:57.303836 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:35:57.304723 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:35:57.304728 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:35:57.306189 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:35:57.306198 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:35:57.307007 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:35:57.307011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:35:57.307113 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:35:57.307136 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 22:35:57.307211 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 22:35:57.307277 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":120,"threads_started":1}
I20260925 22:35:57.307435 8174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:35:57.303177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.195:60435 (local address 127.6.246.194:39623)
0925 22:35:57.303302 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 22:35:57.303307 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:35:57.303400 (+ 93us) server_negotiation.cc:407] Connection header received
0925 22:35:57.303542 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:35:57.303551 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:35:57.303593 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:35:57.303670 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:35:57.303931 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:35:57.304618 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:35:57.306315 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:35:57.306876 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:35:57.307007 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:35:57.307052 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 22:35:57.307233 (+ 181us) server_negotiation.cc:299] Negotiation successful
0925 22:35:57.307322 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 22:35:57.307806 8047 leader_election.cc:302] T 2adf376df4b04dbf915449c89fe638b8 P 9685da8e324f4409a28b0ecc91fc1d48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66d33d841093462b8ffe26045545d6e6, 9685da8e324f4409a28b0ecc91fc1d48; no voters:
I20260925 22:35:57.308066 8182 raft_consensus.cc:2781] T 2adf376df4b04dbf915449c89fe638b8 P 9685da8e324f4409a28b0ecc91fc1d48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:35:57.308063 7980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2adf376df4b04dbf915449c89fe638b8" candidate_uuid: "9685da8e324f4409a28b0ecc91fc1d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea6625f0168438d8efdf01b9ccede02" is_pre_election: true
I20260925 22:35:57.308142 8182 raft_consensus.cc:482] T 2adf376df4b04dbf915449c89fe638b8 P 9685da8e324f4409a28b0ecc91fc1d48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:35:57.308173 8182 raft_consensus.cc:3037] T 2adf376df4b04dbf915449c89fe638b8 P 9685da8e324f4409a28b0ecc91fc1d48 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:35:57.308318 7980 raft_consensus.cc:2454] T 2adf376df4b04dbf915449c89fe638b8 P cea6625f0168438d8efdf01b9ccede02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9685da8e324f4409a28b0ecc91fc1d48 in term 0.
I20260925 22:35:57.309382 8182 raft_consensus.cc:504] T 2adf376df4b04dbf915449c89fe638b8 P 9685da8e324f4409a28b0ecc91fc1d48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9685da8e324f4409a28b0ecc91fc1d48" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 39097 } } peers { permanent_uuid: "66d33d841093462b8ffe26045545d6e6" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 46147 } } peers { permanent_uuid: "cea6625f0168438d8efdf01b9ccede02" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 39623 } }
I20260925 22:35:57.309731 8182 leader_election.cc:288] T 2adf376df4b04dbf915449c89fe638b8 P 9685da8e324f4409a28b0ecc91fc1d48 [CANDIDATE]: Term 1 election: Requested vote from peers 66d33d841093462b8ffe26045545d6e6 (127.6.246.193:46147), cea6625f0168438d8efdf01b9ccede02 (127.6.246.194:39623)
I20260925 22:35:57.310070 7980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2adf376df4b04dbf915449c89fe638b8" candidate_uuid: "9685da8e324f4409a28b0ecc91fc1d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea6625f0168438d8efdf01b9ccede02"
I20260925 22:35:57.310070 7846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2adf376df4b04dbf915449c89fe638b8" candidate_uuid: "9685da8e324f4409a28b0ecc91fc1d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66d33d841093462b8ffe26045545d6e6"
I20260925 22:35:57.310181 7980 raft_consensus.cc:3037] T 2adf376df4b04dbf915449c89fe638b8 P cea6625f0168438d8efdf01b9ccede02 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:35:57.310182 7846 raft_consensus.cc:3037] T 2adf376df4b04dbf915449c89fe638b8 P 66d33d841093462b8ffe26045545d6e6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:35:57.311403 7846 raft_consensus.cc:2454] T 2adf376df4b04dbf915449c89fe638b8 P 66d33d841093462b8ffe26045545d6e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9685da8e324f4409a28b0ecc91fc1d48 in term 1.
I20260925 22:35:57.311404 7980 raft_consensus.cc:2454] T 2adf376df4b04dbf915449c89fe638b8 P cea6625f0168438d8efdf01b9ccede02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9685da8e324f4409a28b0ecc91fc1d48 in term 1.
I20260925 22:35:57.311709 8047 leader_election.cc:302] T 2adf376df4b04dbf915449c89fe638b8 P 9685da8e324f4409a28b0ecc91fc1d48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66d33d841093462b8ffe26045545d6e6, 9685da8e324f4409a28b0ecc91fc1d48; no voters:
I20260925 22:35:57.311875 8182 raft_consensus.cc:2781] T 2adf376df4b04dbf915449c89fe638b8 P 9685da8e324f4409a28b0ecc91fc1d48 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:35:57.312101 8182 raft_consensus.cc:686] T 2adf376df4b04dbf915449c89fe638b8 P 9685da8e324f4409a28b0ecc91fc1d48 [term 1 LEADER]: Becoming Leader. State: Replica: 9685da8e324f4409a28b0ecc91fc1d48, State: Running, Role: LEADER
I20260925 22:35:57.312413 8182 consensus_queue.cc:237] T 2adf376df4b04dbf915449c89fe638b8 P 9685da8e324f4409a28b0ecc91fc1d48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9685da8e324f4409a28b0ecc91fc1d48" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 39097 } } peers { permanent_uuid: "66d33d841093462b8ffe26045545d6e6" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 46147 } } peers { permanent_uuid: "cea6625f0168438d8efdf01b9ccede02" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 39623 } }
I20260925 22:35:57.315408 7701 catalog_manager.cc:5495] T 2adf376df4b04dbf915449c89fe638b8 P 9685da8e324f4409a28b0ecc91fc1d48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9685da8e324f4409a28b0ecc91fc1d48 (127.6.246.195). New cstate: current_term: 1 leader_uuid: "9685da8e324f4409a28b0ecc91fc1d48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9685da8e324f4409a28b0ecc91fc1d48" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 39097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66d33d841093462b8ffe26045545d6e6" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 46147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cea6625f0168438d8efdf01b9ccede02" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 39623 } health_report { overall_health: UNKNOWN } } }
Sep 25 22:35:57 dist-test-slave-wjdr krb5kdc[7656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.246.193@KRBTEST.COM
I20260925 22:35:57.327962 8176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:35:57.319296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55986 (local address 127.6.246.193:46147)
0925 22:35:57.319465 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 22:35:57.319471 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:35:57.319655 (+ 184us) server_negotiation.cc:407] Connection header received
0925 22:35:57.319804 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:35:57.319807 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:35:57.319862 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:35:57.319946 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:35:57.320702 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:35:57.321259 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:35:57.322084 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:35:57.322309 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:35:57.322421 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:35:57.324814 (+ 2393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:35:57.324823 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:35:57.324835 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:35:57.324862 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:35:57.326470 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:35:57.326941 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:35:57.326946 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:35:57.326947 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:35:57.327020 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:35:57.327246 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:35:57.327248 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:35:57.327249 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:35:57.327457 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:35:57.327605 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 22:35:57.327742 (+ 137us) server_negotiation.cc:299] Negotiation successful
0925 22:35:57.327828 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Sep 25 22:35:57 dist-test-slave-wjdr krb5kdc[7656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375757, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 22:35:57 dist-test-slave-wjdr krb5kdc[7656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375757, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.246.193@KRBTEST.COM
I20260925 22:35:57.353868 8176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:35:57.345556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55994 (local address 127.6.246.193:46147)
0925 22:35:57.345714 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 22:35:57.345720 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:35:57.345849 (+ 129us) server_negotiation.cc:407] Connection header received
0925 22:35:57.346007 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:35:57.346010 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:35:57.346059 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:35:57.346144 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:35:57.346775 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:35:57.347416 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:35:57.348259 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:35:57.348624 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:35:57.348730 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:35:57.351074 (+ 2344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:35:57.351076 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:35:57.351079 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:35:57.351100 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:35:57.352343 (+ 1243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:35:57.352774 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:35:57.352777 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:35:57.352778 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:35:57.352857 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:35:57.353094 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:35:57.353097 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:35:57.353098 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:35:57.353258 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:35:57.353360 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 22:35:57.353581 (+ 221us) server_negotiation.cc:299] Negotiation successful
0925 22:35:57.353679 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
Sep 25 22:35:57 dist-test-slave-wjdr krb5kdc[7656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375757, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 22:35:57 dist-test-slave-wjdr krb5kdc[7656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375757, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.246.193@KRBTEST.COM
I20260925 22:35:57.378676 8176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:35:57.370540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55998 (local address 127.6.246.193:46147)
0925 22:35:57.370708 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 22:35:57.370713 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:35:57.370788 (+ 75us) server_negotiation.cc:407] Connection header received
0925 22:35:57.370897 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:35:57.370900 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:35:57.370939 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:35:57.371007 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:35:57.371631 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:35:57.372131 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:35:57.372977 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:35:57.373177 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:35:57.373270 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:35:57.375665 (+ 2395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:35:57.375668 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:35:57.375670 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:35:57.375691 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:35:57.377167 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:35:57.377591 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:35:57.377593 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:35:57.377595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:35:57.377671 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:35:57.377913 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:35:57.377916 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:35:57.377917 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:35:57.378066 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:35:57.378164 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 22:35:57.378370 (+ 206us) server_negotiation.cc:299] Negotiation successful
0925 22:35:57.378462 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260925 22:35:57.380013 7806 tablet_service.cc:2908] Rejecting scan request for tablet 2adf376df4b04dbf915449c89fe638b8: Uninitialized: clean time has not yet been initialized
I20260925 22:35:57.381927 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 7763
I20260925 22:35:57.387682 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 7897
I20260925 22:35:57.393134 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 8031
I20260925 22:35:57.398181 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 7669
2026-09-25T22:35:57Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2382 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:35:57 dist-test-slave-wjdr krb5kdc[8201](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:35:57 dist-test-slave-wjdr krb5kdc[8201](info): set up 2 sockets
Sep 25 22:35:57 dist-test-slave-wjdr krb5kdc[8201](info): commencing operation
krb5kdc: starting...
W20260925 22:35:58.464551 7131 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Sep 25 22:35:58 dist-test-slave-wjdr krb5kdc[8201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:35:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:35:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.246.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab.
Entry for principal kudu/127.6.246.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab.
I20260925 22:35:58.583604 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:44481
--webserver_interface=127.6.246.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab
--principal=kudu/127.6.246.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:39539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.246.254:44481
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:35:58.691936 8214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:35:58.692214 8214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:35:58.692287 8214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:35:58.696276 8214 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:35:58.696362 8214 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 22:35:58.696391 8214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:35:58.696432 8214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:35:58.696491 8214 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:35:58.696537 8214 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:35:58.701639 8214 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:39539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.246.254:44481
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab
--principal=kudu/127.6.246.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:44481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.246.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8214
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:35:58.702693 8214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:35:58.703709 8214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:35:58.711923 8222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:35:58.711941 8219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:35:58.711978 8220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:35:58.712205 8214 server_base.cc:1034] running on GCE node
I20260925 22:35:58.712915 8214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:35:58.713979 8214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:35:58.715143 8214 hybrid_clock.cc:648] HybridClock initialized: now 1790375758715115 us; error 39 us; skew 500 ppm
Sep 25 22:35:58 dist-test-slave-wjdr krb5kdc[8201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375758, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:35:58.721565 8214 init.cc:377] Logged in from keytab as kudu/127.6.246.254@KRBTEST.COM (short username kudu)
I20260925 22:35:58.722527 8214 webserver.cc:460] Webserver started at http://127.6.246.254:42009/ using document root <none> and password file <none>
I20260925 22:35:58.723062 8214 fs_manager.cc:359] Metadata directory not provided
I20260925 22:35:58.723121 8214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:35:58.723342 8214 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:35:58.725456 8214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/minicluster-data/master-0/data/instance:
uuid: "a097ad9e7471412b897b9d2feefb6d8a"
format_stamp: "Formatted at 2026-09-25 22:35:58 on dist-test-slave-wjdr"
server_key: "f953a91bafa748e2e635c7e84207c657"
server_key_iv: "f1df30775c88ae5c1037335e8207861d"
server_key_version: "encryptionkey@0"
I20260925 22:35:58.726027 8214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/minicluster-data/master-0/wal/instance:
uuid: "a097ad9e7471412b897b9d2feefb6d8a"
format_stamp: "Formatted at 2026-09-25 22:35:58 on dist-test-slave-wjdr"
server_key: "f953a91bafa748e2e635c7e84207c657"
server_key_iv: "f1df30775c88ae5c1037335e8207861d"
server_key_version: "encryptionkey@0"
I20260925 22:35:58.729557 8214 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260925 22:35:58.731863 8229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:35:58.733043 8214 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 22:35:58.733163 8214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/minicluster-data/master-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/minicluster-data/master-0/wal
uuid: "a097ad9e7471412b897b9d2feefb6d8a"
format_stamp: "Formatted at 2026-09-25 22:35:58 on dist-test-slave-wjdr"
server_key: "f953a91bafa748e2e635c7e84207c657"
server_key_iv: "f1df30775c88ae5c1037335e8207861d"
server_key_version: "encryptionkey@0"
I20260925 22:35:58.733260 8214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:35:58.745584 8214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:35:58.748217 8214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:35:58.748413 8214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:35:58.756109 8214 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.254:44481
I20260925 22:35:58.756131 8291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.254:44481 every 8 connection(s)
I20260925 22:35:58.757112 8214 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
I20260925 22:35:58.758926 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 8214
I20260925 22:35:58.759055 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/minicluster-data/master-0/wal/instance
I20260925 22:35:58.759385 7131 external_mini_cluster.cc:1428] Setting key d3798331858d62c8cc1fedc2682dec7d
I20260925 22:35:58.760150 8292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:35:58.766191 8292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a097ad9e7471412b897b9d2feefb6d8a: Bootstrap starting.
Sep 25 22:35:58 dist-test-slave-wjdr krb5kdc[8201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
I20260925 22:35:58.768676 8292 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a097ad9e7471412b897b9d2feefb6d8a: Neither blocks nor log segments found. Creating new log.
I20260925 22:35:58.769936 8292 log.cc:824] T 00000000000000000000000000000000 P a097ad9e7471412b897b9d2feefb6d8a: Log is configured to *not* fsync() on all Append() calls
I20260925 22:35:58.771721 8292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a097ad9e7471412b897b9d2feefb6d8a: No bootstrap required, opened a new log
I20260925 22:35:58.773101 8295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:35:58.760797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43260 (local address 127.6.246.254:44481)
0925 22:35:58.761201 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0925 22:35:58.761215 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 22:35:58.761248 (+ 33us) server_negotiation.cc:407] Connection header received
0925 22:35:58.761908 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:35:58.761925 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:35:58.762236 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:35:58.762564 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:35:58.763321 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:35:58.764319 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:35:58.765175 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:35:58.765647 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:35:58.765829 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:35:58.768533 (+ 2704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:35:58.768539 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:35:58.768552 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:35:58.768579 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:35:58.770397 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:35:58.770849 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:35:58.770854 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:35:58.770858 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:35:58.770962 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:35:58.771254 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:35:58.771258 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:35:58.771260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:35:58.771626 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:35:58.771842 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0925 22:35:58.772069 (+ 227us) server_negotiation.cc:299] Negotiation successful
0925 22:35:58.772338 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":131,"threads_started":1}
I20260925 22:35:58.774842 8292 raft_consensus.cc:348] T 00000000000000000000000000000000 P a097ad9e7471412b897b9d2feefb6d8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a097ad9e7471412b897b9d2feefb6d8a" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 44481 } }
I20260925 22:35:58.775075 8292 raft_consensus.cc:374] T 00000000000000000000000000000000 P a097ad9e7471412b897b9d2feefb6d8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:35:58.775177 8292 raft_consensus.cc:729] T 00000000000000000000000000000000 P a097ad9e7471412b897b9d2feefb6d8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a097ad9e7471412b897b9d2feefb6d8a, State: Initialized, Role: FOLLOWER
I20260925 22:35:58.775563 8292 consensus_queue.cc:260] T 00000000000000000000000000000000 P a097ad9e7471412b897b9d2feefb6d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a097ad9e7471412b897b9d2feefb6d8a" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 44481 } }
I20260925 22:35:58.775687 8292 raft_consensus.cc:388] T 00000000000000000000000000000000 P a097ad9e7471412b897b9d2feefb6d8a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:35:58.775732 8292 raft_consensus.cc:482] T 00000000000000000000000000000000 P a097ad9e7471412b897b9d2feefb6d8a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:35:58.775812 8292 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a097ad9e7471412b897b9d2feefb6d8a [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:35:58.776706 8292 raft_consensus.cc:504] T 00000000000000000000000000000000 P a097ad9e7471412b897b9d2feefb6d8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a097ad9e7471412b897b9d2feefb6d8a" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 44481 } }
I20260925 22:35:58.777001 8292 leader_election.cc:302] T 00000000000000000000000000000000 P a097ad9e7471412b897b9d2feefb6d8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a097ad9e7471412b897b9d2feefb6d8a; no voters:
I20260925 22:35:58.777271 8292 leader_election.cc:288] T 00000000000000000000000000000000 P a097ad9e7471412b897b9d2feefb6d8a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:35:58.777357 8297 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a097ad9e7471412b897b9d2feefb6d8a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:35:58.777801 8297 raft_consensus.cc:686] T 00000000000000000000000000000000 P a097ad9e7471412b897b9d2feefb6d8a [term 1 LEADER]: Becoming Leader. State: Replica: a097ad9e7471412b897b9d2feefb6d8a, State: Running, Role: LEADER
I20260925 22:35:58.778136 8297 consensus_queue.cc:237] T 00000000000000000000000000000000 P a097ad9e7471412b897b9d2feefb6d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a097ad9e7471412b897b9d2feefb6d8a" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 44481 } }
I20260925 22:35:58.778514 8292 sys_catalog.cc:564] T 00000000000000000000000000000000 P a097ad9e7471412b897b9d2feefb6d8a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:35:58.779992 8298 sys_catalog.cc:455] T 00000000000000000000000000000000 P a097ad9e7471412b897b9d2feefb6d8a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a097ad9e7471412b897b9d2feefb6d8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a097ad9e7471412b897b9d2feefb6d8a" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 44481 } } }
I20260925 22:35:58.780009 8299 sys_catalog.cc:455] T 00000000000000000000000000000000 P a097ad9e7471412b897b9d2feefb6d8a [sys.catalog]: SysCatalogTable state changed. Reason: New leader a097ad9e7471412b897b9d2feefb6d8a. Latest consensus state: current_term: 1 leader_uuid: "a097ad9e7471412b897b9d2feefb6d8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a097ad9e7471412b897b9d2feefb6d8a" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 44481 } } }
I20260925 22:35:58.780119 8298 sys_catalog.cc:458] T 00000000000000000000000000000000 P a097ad9e7471412b897b9d2feefb6d8a [sys.catalog]: This master's current role is: LEADER
I20260925 22:35:58.780131 8299 sys_catalog.cc:458] T 00000000000000000000000000000000 P a097ad9e7471412b897b9d2feefb6d8a [sys.catalog]: This master's current role is: LEADER
I20260925 22:35:58.780659 8307 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:35:58.783269 8307 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:35:58.788573 8307 catalog_manager.cc:1347] Generated new cluster ID: 5f2a80a13bda42efacd8357e8e5f6d27
I20260925 22:35:58.788638 8307 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:35:58.809199 8307 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:35:58.810127 8307 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:35:58.817680 8307 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a097ad9e7471412b897b9d2feefb6d8a: Generated new TSK 0
I20260925 22:35:58.818567 8307 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.246.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.193.keytab.
Entry for principal kudu/127.6.246.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.193.keytab.
I20260925 22:35:58.855757 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.193:0
--local_ip_for_outbound_sockets=127.6.246.193
--webserver_interface=127.6.246.193
--webserver_port=0
--tserver_master_addrs=127.6.246.254:44481
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.193.keytab
--principal=kudu/127.6.246.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:39539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:35:58.962950 8318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:35:58.963218 8318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:35:58.963292 8318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:35:58.967248 8318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:35:58.967329 8318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:35:58.967399 8318 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 22:35:58.967478 8318 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.193
I20260925 22:35:58.972647 8318 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:39539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.193.keytab
--principal=kudu/127.6.246.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.246.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.246.254:44481
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8318
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.193
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:35:58.973631 8318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:35:58.974606 8318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:35:58.983805 8326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:35:58.983986 8318 server_base.cc:1034] running on GCE node
W20260925 22:35:58.983984 8324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:35:58.983819 8323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:35:58.984527 8318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:35:58.985114 8318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:35:58.986302 8318 hybrid_clock.cc:648] HybridClock initialized: now 1790375758986285 us; error 34 us; skew 500 ppm
Sep 25 22:35:58 dist-test-slave-wjdr krb5kdc[8201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375758, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:35:58.988976 8318 init.cc:377] Logged in from keytab as kudu/127.6.246.193@KRBTEST.COM (short username kudu)
I20260925 22:35:58.989777 8318 webserver.cc:460] Webserver started at http://127.6.246.193:37031/ using document root <none> and password file <none>
I20260925 22:35:58.990337 8318 fs_manager.cc:359] Metadata directory not provided
I20260925 22:35:58.990424 8318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:35:58.990640 8318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:35:58.992635 8318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/minicluster-data/ts-0/data/instance:
uuid: "c088345e1e7f4a43a101bc3ebe83172a"
format_stamp: "Formatted at 2026-09-25 22:35:58 on dist-test-slave-wjdr"
server_key: "6582aca80a0029a5885cd40f313303d7"
server_key_iv: "5071b5a40a61490233f21afd2128d3b4"
server_key_version: "encryptionkey@0"
I20260925 22:35:58.993180 8318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/minicluster-data/ts-0/wal/instance:
uuid: "c088345e1e7f4a43a101bc3ebe83172a"
format_stamp: "Formatted at 2026-09-25 22:35:58 on dist-test-slave-wjdr"
server_key: "6582aca80a0029a5885cd40f313303d7"
server_key_iv: "5071b5a40a61490233f21afd2128d3b4"
server_key_version: "encryptionkey@0"
I20260925 22:35:58.996799 8318 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 22:35:58.999110 8333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:35:59.000355 8318 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:35:59.000509 8318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/minicluster-data/ts-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/minicluster-data/ts-0/wal
uuid: "c088345e1e7f4a43a101bc3ebe83172a"
format_stamp: "Formatted at 2026-09-25 22:35:58 on dist-test-slave-wjdr"
server_key: "6582aca80a0029a5885cd40f313303d7"
server_key_iv: "5071b5a40a61490233f21afd2128d3b4"
server_key_version: "encryptionkey@0"
I20260925 22:35:59.000623 8318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:35:59.017777 8318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:35:59.021027 8318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:35:59.021317 8318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:35:59.022854 8318 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:35:59.022933 8318 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:35:59.023000 8318 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:35:59.023039 8318 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 22:35:59 dist-test-slave-wjdr krb5kdc[8201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375758, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.193@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
I20260925 22:35:59.033865 8318 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.193:46377
I20260925 22:35:59.033918 8448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.193:46377 every 8 connection(s)
I20260925 22:35:59.034864 8318 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
I20260925 22:35:59.037959 8295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:35:59.023761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.193:36939 (local address 127.6.246.254:44481)
0925 22:35:59.023955 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 22:35:59.023961 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:35:59.024810 (+ 849us) server_negotiation.cc:407] Connection header received
0925 22:35:59.025848 (+ 1038us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:35:59.025852 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:35:59.025908 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:35:59.025967 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:35:59.027902 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:35:59.028830 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:35:59.030756 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:35:59.030994 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:35:59.031110 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:35:59.034204 (+ 3094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:35:59.034208 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:35:59.034211 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:35:59.034245 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:35:59.035880 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:35:59.036371 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:35:59.036373 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:35:59.036374 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:35:59.036612 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:35:59.036933 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:35:59.036938 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:35:59.036942 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:35:59.037176 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:35:59.037263 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 22:35:59.037715 (+ 452us) server_negotiation.cc:299] Negotiation successful
0925 22:35:59.037829 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 22:35:59.038667 8342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:35:59.024166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:44481 (local address 127.6.246.193:36939)
0925 22:35:59.024648 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0925 22:35:59.024698 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 22:35:59.025595 (+ 897us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:35:59.026462 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:35:59.026474 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:35:59.026991 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:35:59.027700 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:35:59.027722 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:35:59.028949 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:35:59.028955 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:35:59.030594 (+ 1639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:35:59.030608 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:35:59.031158 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:35:59.031164 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:35:59.031393 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:35:59.032103 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:35:59.032127 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:35:59.034007 (+ 1880us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:35:59.036010 (+ 2003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:35:59.036015 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:35:59.036028 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:35:59.036288 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:35:59.036690 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:35:59.036693 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:35:59.036694 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:35:59.036842 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:35:59.037273 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:35:59.037278 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:35:59.037488 (+ 210us) client_negotiation.cc:769] Sending connection context
0925 22:35:59.037653 (+ 165us) client_negotiation.cc:240] Negotiation successful
0925 22:35:59.037896 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":131,"threads_started":1}
I20260925 22:35:59.039923 8449 heartbeater.cc:347] Connected to a master server at 127.6.246.254:44481
I20260925 22:35:59.040058 8449 heartbeater.cc:464] Registering TS with master...
I20260925 22:35:59.040616 8449 heartbeater.cc:511] Master 127.6.246.254:44481 requested a full tablet report, sending...
I20260925 22:35:59.041484 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 8318
I20260925 22:35:59.041599 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/minicluster-data/ts-0/wal/instance
I20260925 22:35:59.041887 7131 external_mini_cluster.cc:1428] Setting key 4fa88682202a038fa276fe251b1929fd
I20260925 22:35:59.042183 8245 ts_manager.cc:194] Registered new tserver with Master: c088345e1e7f4a43a101bc3ebe83172a (127.6.246.193:46377)
I20260925 22:35:59.044256 8245 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.246.193@KRBTEST.COM'} at 127.6.246.193:36939
I20260925 22:35:59.050052 8295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:35:59.042355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.193:49483 (local address 127.6.246.254:44481)
0925 22:35:59.042486 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 22:35:59.042491 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:35:59.042689 (+ 198us) server_negotiation.cc:407] Connection header received
0925 22:35:59.042770 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:35:59.042773 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:35:59.042812 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:35:59.042897 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:35:59.043243 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:35:59.044143 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:35:59.045465 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:35:59.045680 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:35:59.045790 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:35:59.046955 (+ 1165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:35:59.046958 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:35:59.046960 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:35:59.046980 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:35:59.048396 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:35:59.048939 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:35:59.048942 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:35:59.048943 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:35:59.049011 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:35:59.049313 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:35:59.049315 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:35:59.049316 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:35:59.049459 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:35:59.049529 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 22:35:59.049856 (+ 327us) server_negotiation.cc:299] Negotiation successful
0925 22:35:59.049947 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":90,"server-negotiator.queue_time_us":56}
I20260925 22:35:59.050130 8455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:35:59.042270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:44481 (local address 127.6.246.193:49483)
0925 22:35:59.042610 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0925 22:35:59.042622 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:35:59.042696 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:35:59.042921 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:35:59.042923 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:35:59.043062 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:35:59.043148 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:35:59.043153 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:35:59.044279 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:35:59.044286 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:35:59.045306 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:35:59.045320 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:35:59.045795 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:35:59.045798 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:35:59.045900 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:35:59.046369 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:35:59.046382 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:35:59.046832 (+ 450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:35:59.048543 (+ 1711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:35:59.048549 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:35:59.048552 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:35:59.048839 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:35:59.049103 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:35:59.049108 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:35:59.049112 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:35:59.049213 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:35:59.049549 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:35:59.049552 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:35:59.049651 (+ 99us) client_negotiation.cc:769] Sending connection context
0925 22:35:59.049769 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 22:35:59.049920 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"mutex_wait_us":2,"thread_start_us":109,"threads_started":1}
WARNING: no policy specified for kudu/127.6.246.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.194.keytab.
Entry for principal kudu/127.6.246.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.194.keytab.
I20260925 22:35:59.069833 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.194:0
--local_ip_for_outbound_sockets=127.6.246.194
--webserver_interface=127.6.246.194
--webserver_port=0
--tserver_master_addrs=127.6.246.254:44481
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.194.keytab
--principal=kudu/127.6.246.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:39539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:35:59.178560 8458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:35:59.178861 8458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:35:59.178936 8458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:35:59.183280 8458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:35:59.183370 8458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:35:59.183444 8458 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 22:35:59.183522 8458 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.194
I20260925 22:35:59.188755 8458 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:39539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.194.keytab
--principal=kudu/127.6.246.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.246.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.246.254:44481
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8458
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.194
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:35:59.189862 8458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:35:59.190881 8458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:35:59.200444 8466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:35:59.200440 8463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:35:59.200479 8464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:35:59.200735 8458 server_base.cc:1034] running on GCE node
I20260925 22:35:59.201184 8458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:35:59.201792 8458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:35:59.202966 8458 hybrid_clock.cc:648] HybridClock initialized: now 1790375759202952 us; error 32 us; skew 500 ppm
Sep 25 22:35:59 dist-test-slave-wjdr krb5kdc[8201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375759, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:35:59.205704 8458 init.cc:377] Logged in from keytab as kudu/127.6.246.194@KRBTEST.COM (short username kudu)
I20260925 22:35:59.206566 8458 webserver.cc:460] Webserver started at http://127.6.246.194:35317/ using document root <none> and password file <none>
I20260925 22:35:59.207103 8458 fs_manager.cc:359] Metadata directory not provided
I20260925 22:35:59.207188 8458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:35:59.207398 8458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:35:59.209398 8458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/minicluster-data/ts-1/data/instance:
uuid: "83b39ca9a619489abe2c938e6047a01a"
format_stamp: "Formatted at 2026-09-25 22:35:59 on dist-test-slave-wjdr"
server_key: "47564f55f58bf9502f44ff05028c185d"
server_key_iv: "959d88bf7f9fbe0b1e62774978d688da"
server_key_version: "encryptionkey@0"
I20260925 22:35:59.209932 8458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/minicluster-data/ts-1/wal/instance:
uuid: "83b39ca9a619489abe2c938e6047a01a"
format_stamp: "Formatted at 2026-09-25 22:35:59 on dist-test-slave-wjdr"
server_key: "47564f55f58bf9502f44ff05028c185d"
server_key_iv: "959d88bf7f9fbe0b1e62774978d688da"
server_key_version: "encryptionkey@0"
I20260925 22:35:59.213469 8458 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:35:59.215754 8473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:35:59.216929 8458 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:35:59.217052 8458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/minicluster-data/ts-1/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/minicluster-data/ts-1/wal
uuid: "83b39ca9a619489abe2c938e6047a01a"
format_stamp: "Formatted at 2026-09-25 22:35:59 on dist-test-slave-wjdr"
server_key: "47564f55f58bf9502f44ff05028c185d"
server_key_iv: "959d88bf7f9fbe0b1e62774978d688da"
server_key_version: "encryptionkey@0"
I20260925 22:35:59.217149 8458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:35:59.230976 8458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:35:59.233853 8458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:35:59.234035 8458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:35:59.235380 8458 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:35:59.235456 8458 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:35:59.235543 8458 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:35:59.235594 8458 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 22:35:59 dist-test-slave-wjdr krb5kdc[8201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375759, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.194@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
I20260925 22:35:59.246203 8458 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.194:34205
I20260925 22:35:59.246606 8588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.194:34205 every 8 connection(s)
I20260925 22:35:59.247128 8458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
I20260925 22:35:59.249562 8295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:35:59.236260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.194:51351 (local address 127.6.246.254:44481)
0925 22:35:59.236382 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 22:35:59.236385 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:35:59.237415 (+ 1030us) server_negotiation.cc:407] Connection header received
0925 22:35:59.238557 (+ 1142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:35:59.238560 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:35:59.238618 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:35:59.238705 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:35:59.240062 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:35:59.240836 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:35:59.241942 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:35:59.242163 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:35:59.242277 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:35:59.245690 (+ 3413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:35:59.245695 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:35:59.245697 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:35:59.245729 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:35:59.247427 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:35:59.248008 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:35:59.248013 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:35:59.248016 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:35:59.248109 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:35:59.248410 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:35:59.248412 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:35:59.248413 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:35:59.248643 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:35:59.248721 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 22:35:59.249257 (+ 536us) server_negotiation.cc:299] Negotiation successful
0925 22:35:59.249350 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 22:35:59.250442 8482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:35:59.236796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:44481 (local address 127.6.246.194:51351)
0925 22:35:59.237252 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0925 22:35:59.237295 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 22:35:59.238352 (+ 1057us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:35:59.238952 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:35:59.238959 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:35:59.239348 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:35:59.239922 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:35:59.239936 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:35:59.240956 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:35:59.240959 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:35:59.241818 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:35:59.241826 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:35:59.242313 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:35:59.242318 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:35:59.242519 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:35:59.243067 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:35:59.243091 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:35:59.245527 (+ 2436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:35:59.247562 (+ 2035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:35:59.247567 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:35:59.247581 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:35:59.247895 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:35:59.248204 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:35:59.248207 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:35:59.248208 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:35:59.248340 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:35:59.248765 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:35:59.248771 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:35:59.249045 (+ 274us) client_negotiation.cc:769] Sending connection context
0925 22:35:59.249211 (+ 166us) client_negotiation.cc:240] Negotiation successful
0925 22:35:59.249489 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":138,"threads_started":1}
I20260925 22:35:59.251624 8589 heartbeater.cc:347] Connected to a master server at 127.6.246.254:44481
I20260925 22:35:59.251771 8589 heartbeater.cc:464] Registering TS with master...
I20260925 22:35:59.252238 8589 heartbeater.cc:511] Master 127.6.246.254:44481 requested a full tablet report, sending...
I20260925 22:35:59.253229 8246 ts_manager.cc:194] Registered new tserver with Master: 83b39ca9a619489abe2c938e6047a01a (127.6.246.194:34205)
I20260925 22:35:59.254019 8246 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.246.194@KRBTEST.COM'} at 127.6.246.194:51351
I20260925 22:35:59.255785 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 8458
I20260925 22:35:59.255910 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/minicluster-data/ts-1/wal/instance
I20260925 22:35:59.256146 7131 external_mini_cluster.cc:1428] Setting key 6d7c657fdfa1d37a056ed52f28a63277
I20260925 22:35:59.260484 8295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:35:59.253581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.194:40019 (local address 127.6.246.254:44481)
0925 22:35:59.253683 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0925 22:35:59.253688 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:35:59.253925 (+ 237us) server_negotiation.cc:407] Connection header received
0925 22:35:59.254028 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:35:59.254030 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:35:59.254066 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:35:59.254157 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:35:59.254618 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:35:59.255107 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:35:59.255894 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:35:59.256147 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:35:59.256254 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:35:59.257360 (+ 1106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:35:59.257364 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:35:59.257367 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:35:59.257391 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:35:59.258822 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:35:59.259255 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:35:59.259260 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:35:59.259262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:35:59.259341 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:35:59.259633 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:35:59.259637 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:35:59.259639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:35:59.259850 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:35:59.259958 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 22:35:59.260216 (+ 258us) server_negotiation.cc:299] Negotiation successful
0925 22:35:59.260325 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 22:35:59.260496 8595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:35:59.253472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:44481 (local address 127.6.246.194:40019)
0925 22:35:59.253841 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0925 22:35:59.253854 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:35:59.253951 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:35:59.254172 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:35:59.254175 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:35:59.254402 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:35:59.254517 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:35:59.254528 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:35:59.255234 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:35:59.255237 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:35:59.255783 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:35:59.255790 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:35:59.256255 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:35:59.256258 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:35:59.256359 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:35:59.256860 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:35:59.256870 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:35:59.257212 (+ 342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:35:59.258955 (+ 1743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:35:59.258959 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:35:59.258961 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:35:59.259157 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:35:59.259432 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:35:59.259436 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:35:59.259438 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:35:59.259522 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:35:59.259959 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:35:59.259963 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:35:59.260054 (+ 91us) client_negotiation.cc:769] Sending connection context
0925 22:35:59.260152 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 22:35:59.260286 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":91,"threads_started":1}
WARNING: no policy specified for kudu/127.6.246.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.195.keytab.
Entry for principal kudu/127.6.246.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.195.keytab.
I20260925 22:35:59.282967 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.195:0
--local_ip_for_outbound_sockets=127.6.246.195
--webserver_interface=127.6.246.195
--webserver_port=0
--tserver_master_addrs=127.6.246.254:44481
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.195.keytab
--principal=kudu/127.6.246.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:39539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:35:59.388860 8598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:35:59.389128 8598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:35:59.389205 8598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:35:59.393122 8598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:35:59.393196 8598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:35:59.393262 8598 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 22:35:59.393343 8598 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.195
I20260925 22:35:59.398384 8598 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:39539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.195.keytab
--principal=kudu/127.6.246.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.246.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.246.254:44481
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.195
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:35:59.399377 8598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:35:59.400345 8598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:35:59.409288 8603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:35:59.409332 8606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:35:59.409328 8604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:35:59.409626 8598 server_base.cc:1034] running on GCE node
I20260925 22:35:59.410056 8598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:35:59.410610 8598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:35:59.411780 8598 hybrid_clock.cc:648] HybridClock initialized: now 1790375759411765 us; error 28 us; skew 500 ppm
Sep 25 22:35:59 dist-test-slave-wjdr krb5kdc[8201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375759, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:35:59.414520 8598 init.cc:377] Logged in from keytab as kudu/127.6.246.195@KRBTEST.COM (short username kudu)
I20260925 22:35:59.415355 8598 webserver.cc:460] Webserver started at http://127.6.246.195:39355/ using document root <none> and password file <none>
I20260925 22:35:59.415884 8598 fs_manager.cc:359] Metadata directory not provided
I20260925 22:35:59.415970 8598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:35:59.416184 8598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:35:59.418128 8598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/minicluster-data/ts-2/data/instance:
uuid: "8b2ea59d9ae340fc8cee71292815b734"
format_stamp: "Formatted at 2026-09-25 22:35:59 on dist-test-slave-wjdr"
server_key: "ff511f04f2c379c4c3f33c75adde7bd4"
server_key_iv: "43ad8d3b2ed1c8ee0f977b009c6aa689"
server_key_version: "encryptionkey@0"
I20260925 22:35:59.418675 8598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/minicluster-data/ts-2/wal/instance:
uuid: "8b2ea59d9ae340fc8cee71292815b734"
format_stamp: "Formatted at 2026-09-25 22:35:59 on dist-test-slave-wjdr"
server_key: "ff511f04f2c379c4c3f33c75adde7bd4"
server_key_iv: "43ad8d3b2ed1c8ee0f977b009c6aa689"
server_key_version: "encryptionkey@0"
I20260925 22:35:59.422322 8598 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 22:35:59.424676 8613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:35:59.425972 8598 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:35:59.426097 8598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/minicluster-data/ts-2/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/minicluster-data/ts-2/wal
uuid: "8b2ea59d9ae340fc8cee71292815b734"
format_stamp: "Formatted at 2026-09-25 22:35:59 on dist-test-slave-wjdr"
server_key: "ff511f04f2c379c4c3f33c75adde7bd4"
server_key_iv: "43ad8d3b2ed1c8ee0f977b009c6aa689"
server_key_version: "encryptionkey@0"
I20260925 22:35:59.426200 8598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:35:59.447845 8598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:35:59.450721 8598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:35:59.450905 8598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:35:59.452364 8598 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:35:59.452430 8598 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:35:59.452521 8598 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:35:59.452575 8598 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 22:35:59 dist-test-slave-wjdr krb5kdc[8201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375759, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.195@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
I20260925 22:35:59.462868 8598 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.195:34359
I20260925 22:35:59.463152 8728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.195:34359 every 8 connection(s)
I20260925 22:35:59.463929 8598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
I20260925 22:35:59.465929 8295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:35:59.453137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.195:59117 (local address 127.6.246.254:44481)
0925 22:35:59.453296 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 22:35:59.453303 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:35:59.454595 (+ 1292us) server_negotiation.cc:407] Connection header received
0925 22:35:59.455979 (+ 1384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:35:59.455982 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:35:59.456030 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:35:59.456132 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:35:59.457432 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:35:59.457959 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:35:59.459262 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:35:59.459484 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:35:59.459601 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:35:59.461958 (+ 2357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:35:59.461962 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:35:59.461964 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:35:59.461987 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:35:59.463722 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:35:59.464325 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:35:59.464332 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:35:59.464335 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:35:59.464424 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:35:59.464871 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:35:59.464875 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:35:59.464877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:35:59.465062 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:35:59.465165 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 22:35:59.465634 (+ 469us) server_negotiation.cc:299] Negotiation successful
0925 22:35:59.465751 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 22:35:59.466619 8622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:35:59.453980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:44481 (local address 127.6.246.195:59117)
0925 22:35:59.454445 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0925 22:35:59.454498 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 22:35:59.455751 (+ 1253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:35:59.456316 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:35:59.456323 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:35:59.456796 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:35:59.457281 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:35:59.457295 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:35:59.458091 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:35:59.458095 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:35:59.459141 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:35:59.459150 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:35:59.459641 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:35:59.459645 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:35:59.459805 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:35:59.460332 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:35:59.460347 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:35:59.461761 (+ 1414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:35:59.463872 (+ 2111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:35:59.463876 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:35:59.463887 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:35:59.464181 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:35:59.464586 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:35:59.464590 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:35:59.464593 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:35:59.464758 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:35:59.465176 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:35:59.465184 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:35:59.465411 (+ 227us) client_negotiation.cc:769] Sending connection context
0925 22:35:59.465592 (+ 181us) client_negotiation.cc:240] Negotiation successful
0925 22:35:59.465876 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":103,"threads_started":1}
I20260925 22:35:59.467725 8729 heartbeater.cc:347] Connected to a master server at 127.6.246.254:44481
I20260925 22:35:59.467854 8729 heartbeater.cc:464] Registering TS with master...
I20260925 22:35:59.468303 8729 heartbeater.cc:511] Master 127.6.246.254:44481 requested a full tablet report, sending...
I20260925 22:35:59.468734 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 8598
I20260925 22:35:59.468869 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/minicluster-data/ts-2/wal/instance
I20260925 22:35:59.469147 7131 external_mini_cluster.cc:1428] Setting key d57b352ed8e953eee9d9165f87f451fe
I20260925 22:35:59.469360 8246 ts_manager.cc:194] Registered new tserver with Master: 8b2ea59d9ae340fc8cee71292815b734 (127.6.246.195:34359)
I20260925 22:35:59.470080 8246 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.246.195@KRBTEST.COM'} at 127.6.246.195:59117
I20260925 22:35:59.471112 7131 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:35:59.477591 8735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:35:59.469963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:44481 (local address 127.6.246.195:55445)
0925 22:35:59.470394 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0925 22:35:59.470410 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:35:59.470531 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:35:59.470768 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:35:59.470778 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:35:59.470968 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:35:59.471084 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:35:59.471091 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:35:59.472059 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:35:59.472062 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:35:59.472698 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:35:59.472707 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:35:59.473238 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:35:59.473244 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:35:59.473358 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:35:59.473862 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:35:59.473880 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:35:59.474337 (+ 457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:35:59.475998 (+ 1661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:35:59.476002 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:35:59.476005 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:35:59.476235 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:35:59.476579 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:35:59.476582 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:35:59.476583 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:35:59.476655 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:35:59.477083 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:35:59.477085 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:35:59.477156 (+ 71us) client_negotiation.cc:769] Sending connection context
0925 22:35:59.477235 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 22:35:59.477340 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":96,"threads_started":1}
I20260925 22:35:59.477669 8295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:35:59.470295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.195:55445 (local address 127.6.246.254:44481)
0925 22:35:59.470395 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0925 22:35:59.470400 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:35:59.470473 (+ 73us) server_negotiation.cc:407] Connection header received
0925 22:35:59.470630 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:35:59.470633 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:35:59.470676 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:35:59.470754 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:35:59.471203 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:35:59.471932 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:35:59.472803 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:35:59.473128 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:35:59.473208 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:35:59.474468 (+ 1260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:35:59.474471 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:35:59.474473 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:35:59.474492 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:35:59.475889 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:35:59.476349 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:35:59.476352 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:35:59.476354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:35:59.476444 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:35:59.476742 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:35:59.476748 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:35:59.476753 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:35:59.476987 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:35:59.477093 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 22:35:59.477323 (+ 230us) server_negotiation.cc:299] Negotiation successful
0925 22:35:59.477452 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
Sep 25 22:35:59 dist-test-slave-wjdr krb5kdc[8201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375759, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 22:35:59 dist-test-slave-wjdr krb5kdc[8201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375759, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
I20260925 22:35:59.507664 8295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:35:59.498483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43272 (local address 127.6.246.254:44481)
0925 22:35:59.498670 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 22:35:59.498677 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:35:59.498697 (+ 20us) server_negotiation.cc:407] Connection header received
0925 22:35:59.498756 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:35:59.498761 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:35:59.498821 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:35:59.498931 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:35:59.499863 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:35:59.500835 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:35:59.501734 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:35:59.502112 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:35:59.502231 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:35:59.504931 (+ 2700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:35:59.504934 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:35:59.504936 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:35:59.504959 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:35:59.506213 (+ 1254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:35:59.506584 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:35:59.506586 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:35:59.506587 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:35:59.506658 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:35:59.506949 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:35:59.506951 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:35:59.506952 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:35:59.507113 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:35:59.507212 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 22:35:59.507457 (+ 245us) server_negotiation.cc:299] Negotiation successful
0925 22:35:59.507546 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 22:35:59.510800 8246 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43272:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:35:59.513878 8246 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:35:59.527693 8746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:35:59.521762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.194:34205 (local address 127.0.0.1:37146)
0925 22:35:59.522122 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0925 22:35:59.522178 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 22:35:59.522279 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:35:59.523509 (+ 1230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:35:59.523516 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:35:59.523543 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:35:59.523677 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:35:59.523683 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:35:59.524833 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:35:59.524837 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:35:59.526147 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:35:59.526157 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:35:59.527252 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:35:59.527255 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:35:59.527326 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:35:59.527343 (+ 17us) client_negotiation.cc:769] Sending connection context
0925 22:35:59.527387 (+ 44us) client_negotiation.cc:240] Negotiation successful
0925 22:35:59.527540 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"mutex_wait_us":6,"thread_start_us":118,"threads_started":1}
I20260925 22:35:59.527693 8750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:35:59.522580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50422 (local address 127.6.246.195:34359)
0925 22:35:59.523036 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0925 22:35:59.523047 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 22:35:59.523064 (+ 17us) server_negotiation.cc:407] Connection header received
0925 22:35:59.523124 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:35:59.523128 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:35:59.523297 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:35:59.523454 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:35:59.523852 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:35:59.525038 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:35:59.526380 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:35:59.527004 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:35:59.527110 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:35:59.527263 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 22:35:59.527449 (+ 186us) server_negotiation.cc:299] Negotiation successful
0925 22:35:59.527542 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":89,"threads_started":1}
I20260925 22:35:59.528087 8747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:35:59.521885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37146 (local address 127.6.246.194:34205)
0925 22:35:59.522984 (+ 1099us) server_negotiation.cc:206] Beginning negotiation
0925 22:35:59.522996 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 22:35:59.523018 (+ 22us) server_negotiation.cc:407] Connection header received
0925 22:35:59.523095 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:35:59.523100 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:35:59.523298 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:35:59.523475 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:35:59.523806 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:35:59.524723 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:35:59.526287 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:35:59.527128 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:35:59.527628 (+ 500us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:35:59.527785 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 22:35:59.527852 (+ 67us) server_negotiation.cc:299] Negotiation successful
0925 22:35:59.527950 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":995,"thread_start_us":113,"threads_started":1}
I20260925 22:35:59.528112 8748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:35:59.522039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.195:34359 (local address 127.0.0.1:50422)
0925 22:35:59.522636 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0925 22:35:59.522651 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:35:59.522730 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:35:59.523596 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:35:59.523599 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:35:59.523622 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:35:59.523693 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:35:59.523697 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:35:59.525154 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:35:59.525157 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:35:59.526258 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:35:59.526267 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:35:59.527125 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:35:59.527128 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:35:59.527250 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:35:59.527292 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 22:35:59.527410 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 22:35:59.527540 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"mutex_wait_us":41,"thread_start_us":77,"threads_started":1}
I20260925 22:35:59.528568 8749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:35:59.522518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.193:46377 (local address 127.0.0.1:56476)
0925 22:35:59.522730 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0925 22:35:59.522741 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 22:35:59.522819 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:35:59.523923 (+ 1104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:35:59.523925 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:35:59.523942 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:35:59.524027 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:35:59.524032 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:35:59.525483 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:35:59.525490 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:35:59.526833 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:35:59.526845 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:35:59.528164 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:35:59.528167 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:35:59.528264 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:35:59.528285 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 22:35:59.528356 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 22:35:59.528422 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":87,"threads_started":1}
I20260925 22:35:59.528627 8751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:35:59.523166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56476 (local address 127.6.246.193:46377)
0925 22:35:59.523422 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0925 22:35:59.523494 (+ 72us) server_negotiation.cc:399] Waiting for connection header
0925 22:35:59.523518 (+ 24us) server_negotiation.cc:407] Connection header received
0925 22:35:59.523587 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:35:59.523591 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:35:59.523784 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:35:59.523939 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:35:59.524120 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:35:59.525209 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:35:59.526962 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:35:59.527703 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:35:59.528166 (+ 463us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:35:59.528332 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 22:35:59.528383 (+ 51us) server_negotiation.cc:299] Negotiation successful
0925 22:35:59.528492 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":84,"threads_started":1}
I20260925 22:35:59.531059 8369 tablet_service.cc:1467] Processing CreateTablet for tablet db8122cd93344b65b5fef08ad2741be8 (DEFAULT_TABLE table=test-table [id=c22e898293c74a30bf1803509a22ba1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:35:59.531086 8663 tablet_service.cc:1467] Processing CreateTablet for tablet db8122cd93344b65b5fef08ad2741be8 (DEFAULT_TABLE table=test-table [id=c22e898293c74a30bf1803509a22ba1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:35:59.531086 8523 tablet_service.cc:1467] Processing CreateTablet for tablet db8122cd93344b65b5fef08ad2741be8 (DEFAULT_TABLE table=test-table [id=c22e898293c74a30bf1803509a22ba1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:35:59.532001 8369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db8122cd93344b65b5fef08ad2741be8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:35:59.532021 8523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db8122cd93344b65b5fef08ad2741be8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:35:59.532030 8663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db8122cd93344b65b5fef08ad2741be8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:35:59.538532 8752 tablet_bootstrap.cc:492] T db8122cd93344b65b5fef08ad2741be8 P c088345e1e7f4a43a101bc3ebe83172a: Bootstrap starting.
I20260925 22:35:59.540166 8753 tablet_bootstrap.cc:492] T db8122cd93344b65b5fef08ad2741be8 P 8b2ea59d9ae340fc8cee71292815b734: Bootstrap starting.
I20260925 22:35:59.540757 8754 tablet_bootstrap.cc:492] T db8122cd93344b65b5fef08ad2741be8 P 83b39ca9a619489abe2c938e6047a01a: Bootstrap starting.
I20260925 22:35:59.540925 8752 tablet_bootstrap.cc:654] T db8122cd93344b65b5fef08ad2741be8 P c088345e1e7f4a43a101bc3ebe83172a: Neither blocks nor log segments found. Creating new log.
I20260925 22:35:59.541782 8752 log.cc:824] T db8122cd93344b65b5fef08ad2741be8 P c088345e1e7f4a43a101bc3ebe83172a: Log is configured to *not* fsync() on all Append() calls
I20260925 22:35:59.542023 8753 tablet_bootstrap.cc:654] T db8122cd93344b65b5fef08ad2741be8 P 8b2ea59d9ae340fc8cee71292815b734: Neither blocks nor log segments found. Creating new log.
I20260925 22:35:59.542721 8753 log.cc:824] T db8122cd93344b65b5fef08ad2741be8 P 8b2ea59d9ae340fc8cee71292815b734: Log is configured to *not* fsync() on all Append() calls
I20260925 22:35:59.542778 8754 tablet_bootstrap.cc:654] T db8122cd93344b65b5fef08ad2741be8 P 83b39ca9a619489abe2c938e6047a01a: Neither blocks nor log segments found. Creating new log.
I20260925 22:35:59.543553 8754 log.cc:824] T db8122cd93344b65b5fef08ad2741be8 P 83b39ca9a619489abe2c938e6047a01a: Log is configured to *not* fsync() on all Append() calls
I20260925 22:35:59.543872 8752 tablet_bootstrap.cc:492] T db8122cd93344b65b5fef08ad2741be8 P c088345e1e7f4a43a101bc3ebe83172a: No bootstrap required, opened a new log
I20260925 22:35:59.544061 8752 ts_tablet_manager.cc:1397] T db8122cd93344b65b5fef08ad2741be8 P c088345e1e7f4a43a101bc3ebe83172a: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 22:35:59.544590 8753 tablet_bootstrap.cc:492] T db8122cd93344b65b5fef08ad2741be8 P 8b2ea59d9ae340fc8cee71292815b734: No bootstrap required, opened a new log
I20260925 22:35:59.544761 8753 ts_tablet_manager.cc:1397] T db8122cd93344b65b5fef08ad2741be8 P 8b2ea59d9ae340fc8cee71292815b734: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 22:35:59.545356 8754 tablet_bootstrap.cc:492] T db8122cd93344b65b5fef08ad2741be8 P 83b39ca9a619489abe2c938e6047a01a: No bootstrap required, opened a new log
I20260925 22:35:59.545542 8754 ts_tablet_manager.cc:1397] T db8122cd93344b65b5fef08ad2741be8 P 83b39ca9a619489abe2c938e6047a01a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 22:35:59.548161 8754 raft_consensus.cc:348] T db8122cd93344b65b5fef08ad2741be8 P 83b39ca9a619489abe2c938e6047a01a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83b39ca9a619489abe2c938e6047a01a" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 34205 } } peers { permanent_uuid: "c088345e1e7f4a43a101bc3ebe83172a" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 46377 } } peers { permanent_uuid: "8b2ea59d9ae340fc8cee71292815b734" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 34359 } }
I20260925 22:35:59.548094 8752 raft_consensus.cc:348] T db8122cd93344b65b5fef08ad2741be8 P c088345e1e7f4a43a101bc3ebe83172a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83b39ca9a619489abe2c938e6047a01a" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 34205 } } peers { permanent_uuid: "c088345e1e7f4a43a101bc3ebe83172a" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 46377 } } peers { permanent_uuid: "8b2ea59d9ae340fc8cee71292815b734" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 34359 } }
I20260925 22:35:59.548383 8754 raft_consensus.cc:374] T db8122cd93344b65b5fef08ad2741be8 P 83b39ca9a619489abe2c938e6047a01a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:35:59.548384 8752 raft_consensus.cc:374] T db8122cd93344b65b5fef08ad2741be8 P c088345e1e7f4a43a101bc3ebe83172a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:35:59.548447 8752 raft_consensus.cc:729] T db8122cd93344b65b5fef08ad2741be8 P c088345e1e7f4a43a101bc3ebe83172a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c088345e1e7f4a43a101bc3ebe83172a, State: Initialized, Role: FOLLOWER
I20260925 22:35:59.548463 8754 raft_consensus.cc:729] T db8122cd93344b65b5fef08ad2741be8 P 83b39ca9a619489abe2c938e6047a01a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b39ca9a619489abe2c938e6047a01a, State: Initialized, Role: FOLLOWER
I20260925 22:35:59.548383 8753 raft_consensus.cc:348] T db8122cd93344b65b5fef08ad2741be8 P 8b2ea59d9ae340fc8cee71292815b734 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83b39ca9a619489abe2c938e6047a01a" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 34205 } } peers { permanent_uuid: "c088345e1e7f4a43a101bc3ebe83172a" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 46377 } } peers { permanent_uuid: "8b2ea59d9ae340fc8cee71292815b734" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 34359 } }
I20260925 22:35:59.548666 8753 raft_consensus.cc:374] T db8122cd93344b65b5fef08ad2741be8 P 8b2ea59d9ae340fc8cee71292815b734 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:35:59.548743 8753 raft_consensus.cc:729] T db8122cd93344b65b5fef08ad2741be8 P 8b2ea59d9ae340fc8cee71292815b734 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b2ea59d9ae340fc8cee71292815b734, State: Initialized, Role: FOLLOWER
I20260925 22:35:59.548990 8754 consensus_queue.cc:260] T db8122cd93344b65b5fef08ad2741be8 P 83b39ca9a619489abe2c938e6047a01a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83b39ca9a619489abe2c938e6047a01a" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 34205 } } peers { permanent_uuid: "c088345e1e7f4a43a101bc3ebe83172a" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 46377 } } peers { permanent_uuid: "8b2ea59d9ae340fc8cee71292815b734" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 34359 } }
I20260925 22:35:59.548980 8752 consensus_queue.cc:260] T db8122cd93344b65b5fef08ad2741be8 P c088345e1e7f4a43a101bc3ebe83172a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83b39ca9a619489abe2c938e6047a01a" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 34205 } } peers { permanent_uuid: "c088345e1e7f4a43a101bc3ebe83172a" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 46377 } } peers { permanent_uuid: "8b2ea59d9ae340fc8cee71292815b734" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 34359 } }
I20260925 22:35:59.549078 8753 consensus_queue.cc:260] T db8122cd93344b65b5fef08ad2741be8 P 8b2ea59d9ae340fc8cee71292815b734 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83b39ca9a619489abe2c938e6047a01a" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 34205 } } peers { permanent_uuid: "c088345e1e7f4a43a101bc3ebe83172a" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 46377 } } peers { permanent_uuid: "8b2ea59d9ae340fc8cee71292815b734" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 34359 } }
I20260925 22:35:59.549594 8589 heartbeater.cc:503] Master 127.6.246.254:44481 was elected leader, sending a full tablet report...
I20260925 22:35:59.549965 8753 ts_tablet_manager.cc:1428] T db8122cd93344b65b5fef08ad2741be8 P 8b2ea59d9ae340fc8cee71292815b734: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260925 22:35:59.549996 8752 ts_tablet_manager.cc:1428] T db8122cd93344b65b5fef08ad2741be8 P c088345e1e7f4a43a101bc3ebe83172a: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20260925 22:35:59.550312 8729 heartbeater.cc:503] Master 127.6.246.254:44481 was elected leader, sending a full tablet report...
I20260925 22:35:59.551136 8754 ts_tablet_manager.cc:1428] T db8122cd93344b65b5fef08ad2741be8 P 83b39ca9a619489abe2c938e6047a01a: Time spent starting tablet: real 0.006s user 0.001s sys 0.004s
I20260925 22:35:59.551319 8449 heartbeater.cc:503] Master 127.6.246.254:44481 was elected leader, sending a full tablet report...
I20260925 22:35:59.680619 8760 raft_consensus.cc:482] T db8122cd93344b65b5fef08ad2741be8 P 8b2ea59d9ae340fc8cee71292815b734 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:35:59.680873 8760 raft_consensus.cc:504] T db8122cd93344b65b5fef08ad2741be8 P 8b2ea59d9ae340fc8cee71292815b734 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83b39ca9a619489abe2c938e6047a01a" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 34205 } } peers { permanent_uuid: "c088345e1e7f4a43a101bc3ebe83172a" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 46377 } } peers { permanent_uuid: "8b2ea59d9ae340fc8cee71292815b734" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 34359 } }
I20260925 22:35:59.682044 8760 leader_election.cc:288] T db8122cd93344b65b5fef08ad2741be8 P 8b2ea59d9ae340fc8cee71292815b734 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83b39ca9a619489abe2c938e6047a01a (127.6.246.194:34205), c088345e1e7f4a43a101bc3ebe83172a (127.6.246.193:46377)
I20260925 22:35:59.685945 8622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:35:59.682265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.194:34205 (local address 127.6.246.195:40703)
0925 22:35:59.682389 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 22:35:59.682407 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 22:35:59.682503 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:35:59.682759 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:35:59.682762 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:35:59.682780 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:35:59.682921 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:35:59.682928 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:35:59.683871 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:35:59.683875 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:35:59.684813 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:35:59.684821 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:35:59.685647 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:35:59.685649 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:35:59.685723 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:35:59.685740 (+ 17us) client_negotiation.cc:769] Sending connection context
0925 22:35:59.685796 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 22:35:59.685846 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 22:35:59.686030 8747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:35:59.682383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.195:40703 (local address 127.6.246.194:34205)
0925 22:35:59.682518 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 22:35:59.682522 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:35:59.682539 (+ 17us) server_negotiation.cc:407] Connection header received
0925 22:35:59.682599 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:35:59.682602 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:35:59.682660 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:35:59.682728 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:35:59.683044 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:35:59.683776 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:35:59.684931 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:35:59.685540 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:35:59.685657 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:35:59.685707 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 22:35:59.685829 (+ 122us) server_negotiation.cc:299] Negotiation successful
0925 22:35:59.685895 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 22:35:59.686664 8543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db8122cd93344b65b5fef08ad2741be8" candidate_uuid: "8b2ea59d9ae340fc8cee71292815b734" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b39ca9a619489abe2c938e6047a01a" is_pre_election: true
I20260925 22:35:59.687006 8543 raft_consensus.cc:2454] T db8122cd93344b65b5fef08ad2741be8 P 83b39ca9a619489abe2c938e6047a01a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b2ea59d9ae340fc8cee71292815b734 in term 0.
I20260925 22:35:59.687238 8761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:35:59.682539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.193:46377 (local address 127.6.246.195:44799)
0925 22:35:59.682947 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0925 22:35:59.682959 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:35:59.683058 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:35:59.683287 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:35:59.683290 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:35:59.683306 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:35:59.683422 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:35:59.683428 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:35:59.684245 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:35:59.684248 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:35:59.685939 (+ 1691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:35:59.685948 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:35:59.686809 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:35:59.686816 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:35:59.686911 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:35:59.686930 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 22:35:59.687040 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 22:35:59.687104 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":156,"threads_started":1}
I20260925 22:35:59.687227 8751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:35:59.682954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.195:44799 (local address 127.6.246.193:46377)
0925 22:35:59.683088 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 22:35:59.683091 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:35:59.683103 (+ 12us) server_negotiation.cc:407] Connection header received
0925 22:35:59.683144 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:35:59.683146 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:35:59.683180 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:35:59.683236 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:35:59.683532 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:35:59.684129 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:35:59.686077 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:35:59.686609 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:35:59.686730 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:35:59.686785 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 22:35:59.687056 (+ 271us) server_negotiation.cc:299] Negotiation successful
0925 22:35:59.687127 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 22:35:59.687522 8617 leader_election.cc:302] T db8122cd93344b65b5fef08ad2741be8 P 8b2ea59d9ae340fc8cee71292815b734 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b39ca9a619489abe2c938e6047a01a, 8b2ea59d9ae340fc8cee71292815b734; no voters:
I20260925 22:35:59.687784 8760 raft_consensus.cc:2781] T db8122cd93344b65b5fef08ad2741be8 P 8b2ea59d9ae340fc8cee71292815b734 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:35:59.687876 8760 raft_consensus.cc:482] T db8122cd93344b65b5fef08ad2741be8 P 8b2ea59d9ae340fc8cee71292815b734 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:35:59.687933 8760 raft_consensus.cc:3037] T db8122cd93344b65b5fef08ad2741be8 P 8b2ea59d9ae340fc8cee71292815b734 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:35:59.688257 8403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db8122cd93344b65b5fef08ad2741be8" candidate_uuid: "8b2ea59d9ae340fc8cee71292815b734" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c088345e1e7f4a43a101bc3ebe83172a" is_pre_election: true
I20260925 22:35:59.688547 8403 raft_consensus.cc:2454] T db8122cd93344b65b5fef08ad2741be8 P c088345e1e7f4a43a101bc3ebe83172a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b2ea59d9ae340fc8cee71292815b734 in term 0.
I20260925 22:35:59.689302 8760 raft_consensus.cc:504] T db8122cd93344b65b5fef08ad2741be8 P 8b2ea59d9ae340fc8cee71292815b734 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83b39ca9a619489abe2c938e6047a01a" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 34205 } } peers { permanent_uuid: "c088345e1e7f4a43a101bc3ebe83172a" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 46377 } } peers { permanent_uuid: "8b2ea59d9ae340fc8cee71292815b734" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 34359 } }
I20260925 22:35:59.689668 8760 leader_election.cc:288] T db8122cd93344b65b5fef08ad2741be8 P 8b2ea59d9ae340fc8cee71292815b734 [CANDIDATE]: Term 1 election: Requested vote from peers 83b39ca9a619489abe2c938e6047a01a (127.6.246.194:34205), c088345e1e7f4a43a101bc3ebe83172a (127.6.246.193:46377)
I20260925 22:35:59.689967 8543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db8122cd93344b65b5fef08ad2741be8" candidate_uuid: "8b2ea59d9ae340fc8cee71292815b734" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b39ca9a619489abe2c938e6047a01a"
I20260925 22:35:59.690055 8403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db8122cd93344b65b5fef08ad2741be8" candidate_uuid: "8b2ea59d9ae340fc8cee71292815b734" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c088345e1e7f4a43a101bc3ebe83172a"
I20260925 22:35:59.690092 8543 raft_consensus.cc:3037] T db8122cd93344b65b5fef08ad2741be8 P 83b39ca9a619489abe2c938e6047a01a [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:35:59.690177 8403 raft_consensus.cc:3037] T db8122cd93344b65b5fef08ad2741be8 P c088345e1e7f4a43a101bc3ebe83172a [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:35:59.691119 8403 raft_consensus.cc:2454] T db8122cd93344b65b5fef08ad2741be8 P c088345e1e7f4a43a101bc3ebe83172a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b2ea59d9ae340fc8cee71292815b734 in term 1.
I20260925 22:35:59.691390 8617 leader_election.cc:302] T db8122cd93344b65b5fef08ad2741be8 P 8b2ea59d9ae340fc8cee71292815b734 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b2ea59d9ae340fc8cee71292815b734, c088345e1e7f4a43a101bc3ebe83172a; no voters:
I20260925 22:35:59.691530 8760 raft_consensus.cc:2781] T db8122cd93344b65b5fef08ad2741be8 P 8b2ea59d9ae340fc8cee71292815b734 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:35:59.691569 8543 raft_consensus.cc:2454] T db8122cd93344b65b5fef08ad2741be8 P 83b39ca9a619489abe2c938e6047a01a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b2ea59d9ae340fc8cee71292815b734 in term 1.
I20260925 22:35:59.691776 8760 raft_consensus.cc:686] T db8122cd93344b65b5fef08ad2741be8 P 8b2ea59d9ae340fc8cee71292815b734 [term 1 LEADER]: Becoming Leader. State: Replica: 8b2ea59d9ae340fc8cee71292815b734, State: Running, Role: LEADER
I20260925 22:35:59.692166 8760 consensus_queue.cc:237] T db8122cd93344b65b5fef08ad2741be8 P 8b2ea59d9ae340fc8cee71292815b734 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83b39ca9a619489abe2c938e6047a01a" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 34205 } } peers { permanent_uuid: "c088345e1e7f4a43a101bc3ebe83172a" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 46377 } } peers { permanent_uuid: "8b2ea59d9ae340fc8cee71292815b734" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 34359 } }
I20260925 22:35:59.695386 8244 catalog_manager.cc:5495] T db8122cd93344b65b5fef08ad2741be8 P 8b2ea59d9ae340fc8cee71292815b734 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b2ea59d9ae340fc8cee71292815b734 (127.6.246.195). New cstate: current_term: 1 leader_uuid: "8b2ea59d9ae340fc8cee71292815b734" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83b39ca9a619489abe2c938e6047a01a" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 34205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c088345e1e7f4a43a101bc3ebe83172a" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 46377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b2ea59d9ae340fc8cee71292815b734" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 34359 } health_report { overall_health: HEALTHY } } }
W20260925 22:35:59.714856 8730 tablet.cc:2369] T db8122cd93344b65b5fef08ad2741be8 P 8b2ea59d9ae340fc8cee71292815b734: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:35:59.719368 8750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:35:59.715662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50424 (local address 127.6.246.195:34359)
0925 22:35:59.715785 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 22:35:59.715791 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:35:59.715826 (+ 35us) server_negotiation.cc:407] Connection header received
0925 22:35:59.716055 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:35:59.716057 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:35:59.716131 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:35:59.716246 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:35:59.716538 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:35:59.717045 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:35:59.717980 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:35:59.718206 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:35:59.718321 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:35:59.718621 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:35:59.718979 (+ 358us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:35:59.719097 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 22:35:59.719174 (+ 77us) server_negotiation.cc:299] Negotiation successful
0925 22:35:59.719224 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 22:35:59.724653 8403 raft_consensus.cc:1261] T db8122cd93344b65b5fef08ad2741be8 P c088345e1e7f4a43a101bc3ebe83172a [term 1 FOLLOWER]: Refusing update from remote peer 8b2ea59d9ae340fc8cee71292815b734: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:35:59.724730 8543 raft_consensus.cc:1261] T db8122cd93344b65b5fef08ad2741be8 P 83b39ca9a619489abe2c938e6047a01a [term 1 FOLLOWER]: Refusing update from remote peer 8b2ea59d9ae340fc8cee71292815b734: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:35:59.725306 8760 consensus_queue.cc:1035] T db8122cd93344b65b5fef08ad2741be8 P 8b2ea59d9ae340fc8cee71292815b734 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c088345e1e7f4a43a101bc3ebe83172a" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 46377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:35:59.725504 8762 consensus_queue.cc:1035] T db8122cd93344b65b5fef08ad2741be8 P 8b2ea59d9ae340fc8cee71292815b734 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b39ca9a619489abe2c938e6047a01a" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 34205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:35:59.731609 8765 mvcc.cc:205] Tried to move back new op lower bound from 7333379111825661952 to 7333379111701884928. Current Snapshot: MvccSnapshot[applied={T|T < 7333379111825661952}]
I20260925 22:35:59.733141 8766 mvcc.cc:205] Tried to move back new op lower bound from 7333379111825661952 to 7333379111701884928. Current Snapshot: MvccSnapshot[applied={T|T < 7333379111825661952}]
I20260925 22:35:59.733332 8767 mvcc.cc:205] Tried to move back new op lower bound from 7333379111825661952 to 7333379111701884928. Current Snapshot: MvccSnapshot[applied={T|T < 7333379111825661952}]
I20260925 22:35:59.742777 8244 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:35:59.742954 8244 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:35:59.745393 8244 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a097ad9e7471412b897b9d2feefb6d8a: Sending DeleteTablet for 3 replicas of tablet db8122cd93344b65b5fef08ad2741be8
I20260925 22:35:59.746187 8663 tablet_service.cc:1514] Processing DeleteTablet for tablet db8122cd93344b65b5fef08ad2741be8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:35:59 UTC) from {username='kudu', principal='kudu/127.6.246.254@KRBTEST.COM'} at 127.0.0.1:50422
I20260925 22:35:59.746263 8523 tablet_service.cc:1514] Processing DeleteTablet for tablet db8122cd93344b65b5fef08ad2741be8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:35:59 UTC) from {username='kudu', principal='kudu/127.6.246.254@KRBTEST.COM'} at 127.0.0.1:37146
I20260925 22:35:59.746426 8369 tablet_service.cc:1514] Processing DeleteTablet for tablet db8122cd93344b65b5fef08ad2741be8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:35:59 UTC) from {username='kudu', principal='kudu/127.6.246.254@KRBTEST.COM'} at 127.0.0.1:56476
I20260925 22:35:59.746861 8776 tablet_replica.cc:330] T db8122cd93344b65b5fef08ad2741be8 P c088345e1e7f4a43a101bc3ebe83172a: stopping tablet replica
I20260925 22:35:59.746974 8775 tablet_replica.cc:330] T db8122cd93344b65b5fef08ad2741be8 P 83b39ca9a619489abe2c938e6047a01a: stopping tablet replica
I20260925 22:35:59.747229 8775 raft_consensus.cc:2229] T db8122cd93344b65b5fef08ad2741be8 P 83b39ca9a619489abe2c938e6047a01a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:35:59.747231 8776 raft_consensus.cc:2229] T db8122cd93344b65b5fef08ad2741be8 P c088345e1e7f4a43a101bc3ebe83172a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:35:59.747346 8244 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43272:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 22:35:59.747524 8776 raft_consensus.cc:2258] T db8122cd93344b65b5fef08ad2741be8 P c088345e1e7f4a43a101bc3ebe83172a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:35:59.747534 8775 raft_consensus.cc:2258] T db8122cd93344b65b5fef08ad2741be8 P 83b39ca9a619489abe2c938e6047a01a [term 1 FOLLOWER]: Raft consensus is shut down!
W20260925 22:35:59.747745 8244 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:35:59.749269 8775 ts_tablet_manager.cc:1905] T db8122cd93344b65b5fef08ad2741be8 P 83b39ca9a619489abe2c938e6047a01a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:35:59.749269 8776 ts_tablet_manager.cc:1905] T db8122cd93344b65b5fef08ad2741be8 P c088345e1e7f4a43a101bc3ebe83172a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:35:59.752182 8776 ts_tablet_manager.cc:1918] T db8122cd93344b65b5fef08ad2741be8 P c088345e1e7f4a43a101bc3ebe83172a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:35:59.752349 8776 log.cc:1196] T db8122cd93344b65b5fef08ad2741be8 P c088345e1e7f4a43a101bc3ebe83172a: Deleting WAL directory at /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/minicluster-data/ts-0/wal/wals/db8122cd93344b65b5fef08ad2741be8
I20260925 22:35:59.752383 8775 ts_tablet_manager.cc:1918] T db8122cd93344b65b5fef08ad2741be8 P 83b39ca9a619489abe2c938e6047a01a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:35:59.752507 8775 log.cc:1196] T db8122cd93344b65b5fef08ad2741be8 P 83b39ca9a619489abe2c938e6047a01a: Deleting WAL directory at /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/minicluster-data/ts-1/wal/wals/db8122cd93344b65b5fef08ad2741be8
I20260925 22:35:59.752801 8776 ts_tablet_manager.cc:1939] T db8122cd93344b65b5fef08ad2741be8 P c088345e1e7f4a43a101bc3ebe83172a: Deleting consensus metadata
I20260925 22:35:59.752837 8775 ts_tablet_manager.cc:1939] T db8122cd93344b65b5fef08ad2741be8 P 83b39ca9a619489abe2c938e6047a01a: Deleting consensus metadata
I20260925 22:35:59.752928 8369 tablet_service.cc:1467] Processing CreateTablet for tablet ff9252832b7f45cc9076bca9a7ae8e5c (DEFAULT_TABLE table=test-table [id=a60ccddfef974f74ba005e9b88e79093]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:35:59.753242 8523 tablet_service.cc:1467] Processing CreateTablet for tablet ff9252832b7f45cc9076bca9a7ae8e5c (DEFAULT_TABLE table=test-table [id=a60ccddfef974f74ba005e9b88e79093]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:35:59.753463 8523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff9252832b7f45cc9076bca9a7ae8e5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:35:59.753979 8774 tablet_replica.cc:330] T db8122cd93344b65b5fef08ad2741be8 P 8b2ea59d9ae340fc8cee71292815b734: stopping tablet replica
I20260925 22:35:59.754233 8774 raft_consensus.cc:2229] T db8122cd93344b65b5fef08ad2741be8 P 8b2ea59d9ae340fc8cee71292815b734 [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:35:59.754508 8774 raft_consensus.cc:2258] T db8122cd93344b65b5fef08ad2741be8 P 8b2ea59d9ae340fc8cee71292815b734 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:35:59.755198 8369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff9252832b7f45cc9076bca9a7ae8e5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:35:59.755537 8754 tablet_bootstrap.cc:492] T ff9252832b7f45cc9076bca9a7ae8e5c P 83b39ca9a619489abe2c938e6047a01a: Bootstrap starting.
I20260925 22:35:59.756059 8774 ts_tablet_manager.cc:1905] T db8122cd93344b65b5fef08ad2741be8 P 8b2ea59d9ae340fc8cee71292815b734: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:35:59.756657 8233 catalog_manager.cc:4841] TS 83b39ca9a619489abe2c938e6047a01a (127.6.246.194:34205): tablet db8122cd93344b65b5fef08ad2741be8 (table test-table [id=c22e898293c74a30bf1803509a22ba1a]) successfully deleted
I20260925 22:35:59.756906 8233 catalog_manager.cc:4841] TS c088345e1e7f4a43a101bc3ebe83172a (127.6.246.193:46377): tablet db8122cd93344b65b5fef08ad2741be8 (table test-table [id=c22e898293c74a30bf1803509a22ba1a]) successfully deleted
I20260925 22:35:59.756439 8754 tablet_bootstrap.cc:654] T ff9252832b7f45cc9076bca9a7ae8e5c P 83b39ca9a619489abe2c938e6047a01a: Neither blocks nor log segments found. Creating new log.
I20260925 22:35:59.757516 8752 tablet_bootstrap.cc:492] T ff9252832b7f45cc9076bca9a7ae8e5c P c088345e1e7f4a43a101bc3ebe83172a: Bootstrap starting.
I20260925 22:35:59.758114 8754 tablet_bootstrap.cc:492] T ff9252832b7f45cc9076bca9a7ae8e5c P 83b39ca9a619489abe2c938e6047a01a: No bootstrap required, opened a new log
I20260925 22:35:59.758186 8754 ts_tablet_manager.cc:1397] T ff9252832b7f45cc9076bca9a7ae8e5c P 83b39ca9a619489abe2c938e6047a01a: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260925 22:35:59.758414 8752 tablet_bootstrap.cc:654] T ff9252832b7f45cc9076bca9a7ae8e5c P c088345e1e7f4a43a101bc3ebe83172a: Neither blocks nor log segments found. Creating new log.
I20260925 22:35:59.758621 8754 raft_consensus.cc:348] T ff9252832b7f45cc9076bca9a7ae8e5c P 83b39ca9a619489abe2c938e6047a01a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c088345e1e7f4a43a101bc3ebe83172a" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 46377 } } peers { permanent_uuid: "83b39ca9a619489abe2c938e6047a01a" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 34205 } } peers { permanent_uuid: "8b2ea59d9ae340fc8cee71292815b734" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 34359 } }
I20260925 22:35:59.758723 8754 raft_consensus.cc:374] T ff9252832b7f45cc9076bca9a7ae8e5c P 83b39ca9a619489abe2c938e6047a01a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:35:59.758702 8774 ts_tablet_manager.cc:1918] T db8122cd93344b65b5fef08ad2741be8 P 8b2ea59d9ae340fc8cee71292815b734: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:35:59.758751 8754 raft_consensus.cc:729] T ff9252832b7f45cc9076bca9a7ae8e5c P 83b39ca9a619489abe2c938e6047a01a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b39ca9a619489abe2c938e6047a01a, State: Initialized, Role: FOLLOWER
I20260925 22:35:59.758790 8774 log.cc:1196] T db8122cd93344b65b5fef08ad2741be8 P 8b2ea59d9ae340fc8cee71292815b734: Deleting WAL directory at /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/minicluster-data/ts-2/wal/wals/db8122cd93344b65b5fef08ad2741be8
I20260925 22:35:59.758896 8754 consensus_queue.cc:260] T ff9252832b7f45cc9076bca9a7ae8e5c P 83b39ca9a619489abe2c938e6047a01a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c088345e1e7f4a43a101bc3ebe83172a" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 46377 } } peers { permanent_uuid: "83b39ca9a619489abe2c938e6047a01a" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 34205 } } peers { permanent_uuid: "8b2ea59d9ae340fc8cee71292815b734" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 34359 } }
I20260925 22:35:59.759136 8754 ts_tablet_manager.cc:1428] T ff9252832b7f45cc9076bca9a7ae8e5c P 83b39ca9a619489abe2c938e6047a01a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260925 22:35:59.759155 8774 ts_tablet_manager.cc:1939] T db8122cd93344b65b5fef08ad2741be8 P 8b2ea59d9ae340fc8cee71292815b734: Deleting consensus metadata
I20260925 22:35:59.761415 8663 tablet_service.cc:1467] Processing CreateTablet for tablet ff9252832b7f45cc9076bca9a7ae8e5c (DEFAULT_TABLE table=test-table [id=a60ccddfef974f74ba005e9b88e79093]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:35:59.761467 8230 catalog_manager.cc:4841] TS 8b2ea59d9ae340fc8cee71292815b734 (127.6.246.195:34359): tablet db8122cd93344b65b5fef08ad2741be8 (table test-table [id=c22e898293c74a30bf1803509a22ba1a]) successfully deleted
I20260925 22:35:59.761665 8663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff9252832b7f45cc9076bca9a7ae8e5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:35:59.764133 8753 tablet_bootstrap.cc:492] T ff9252832b7f45cc9076bca9a7ae8e5c P 8b2ea59d9ae340fc8cee71292815b734: Bootstrap starting.
I20260925 22:35:59.764151 8752 tablet_bootstrap.cc:492] T ff9252832b7f45cc9076bca9a7ae8e5c P c088345e1e7f4a43a101bc3ebe83172a: No bootstrap required, opened a new log
I20260925 22:35:59.764225 8752 ts_tablet_manager.cc:1397] T ff9252832b7f45cc9076bca9a7ae8e5c P c088345e1e7f4a43a101bc3ebe83172a: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.001s
I20260925 22:35:59.765002 8752 raft_consensus.cc:348] T ff9252832b7f45cc9076bca9a7ae8e5c P c088345e1e7f4a43a101bc3ebe83172a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c088345e1e7f4a43a101bc3ebe83172a" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 46377 } } peers { permanent_uuid: "83b39ca9a619489abe2c938e6047a01a" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 34205 } } peers { permanent_uuid: "8b2ea59d9ae340fc8cee71292815b734" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 34359 } }
I20260925 22:35:59.765137 8753 tablet_bootstrap.cc:654] T ff9252832b7f45cc9076bca9a7ae8e5c P 8b2ea59d9ae340fc8cee71292815b734: Neither blocks nor log segments found. Creating new log.
I20260925 22:35:59.765136 8752 raft_consensus.cc:374] T ff9252832b7f45cc9076bca9a7ae8e5c P c088345e1e7f4a43a101bc3ebe83172a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:35:59.765172 8752 raft_consensus.cc:729] T ff9252832b7f45cc9076bca9a7ae8e5c P c088345e1e7f4a43a101bc3ebe83172a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c088345e1e7f4a43a101bc3ebe83172a, State: Initialized, Role: FOLLOWER
I20260925 22:35:59.765316 8752 consensus_queue.cc:260] T ff9252832b7f45cc9076bca9a7ae8e5c P c088345e1e7f4a43a101bc3ebe83172a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c088345e1e7f4a43a101bc3ebe83172a" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 46377 } } peers { permanent_uuid: "83b39ca9a619489abe2c938e6047a01a" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 34205 } } peers { permanent_uuid: "8b2ea59d9ae340fc8cee71292815b734" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 34359 } }
I20260925 22:35:59.765579 8752 ts_tablet_manager.cc:1428] T ff9252832b7f45cc9076bca9a7ae8e5c P c088345e1e7f4a43a101bc3ebe83172a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260925 22:35:59.766682 8753 tablet_bootstrap.cc:492] T ff9252832b7f45cc9076bca9a7ae8e5c P 8b2ea59d9ae340fc8cee71292815b734: No bootstrap required, opened a new log
I20260925 22:35:59.766754 8753 ts_tablet_manager.cc:1397] T ff9252832b7f45cc9076bca9a7ae8e5c P 8b2ea59d9ae340fc8cee71292815b734: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 22:35:59.767587 8753 raft_consensus.cc:348] T ff9252832b7f45cc9076bca9a7ae8e5c P 8b2ea59d9ae340fc8cee71292815b734 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c088345e1e7f4a43a101bc3ebe83172a" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 46377 } } peers { permanent_uuid: "83b39ca9a619489abe2c938e6047a01a" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 34205 } } peers { permanent_uuid: "8b2ea59d9ae340fc8cee71292815b734" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 34359 } }
I20260925 22:35:59.767683 8753 raft_consensus.cc:374] T ff9252832b7f45cc9076bca9a7ae8e5c P 8b2ea59d9ae340fc8cee71292815b734 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:35:59.767709 8753 raft_consensus.cc:729] T ff9252832b7f45cc9076bca9a7ae8e5c P 8b2ea59d9ae340fc8cee71292815b734 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b2ea59d9ae340fc8cee71292815b734, State: Initialized, Role: FOLLOWER
I20260925 22:35:59.767824 8753 consensus_queue.cc:260] T ff9252832b7f45cc9076bca9a7ae8e5c P 8b2ea59d9ae340fc8cee71292815b734 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c088345e1e7f4a43a101bc3ebe83172a" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 46377 } } peers { permanent_uuid: "83b39ca9a619489abe2c938e6047a01a" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 34205 } } peers { permanent_uuid: "8b2ea59d9ae340fc8cee71292815b734" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 34359 } }
I20260925 22:35:59.768070 8753 ts_tablet_manager.cc:1428] T ff9252832b7f45cc9076bca9a7ae8e5c P 8b2ea59d9ae340fc8cee71292815b734: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20260925 22:35:59.786321 8450 tablet.cc:2369] T ff9252832b7f45cc9076bca9a7ae8e5c P c088345e1e7f4a43a101bc3ebe83172a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 22:35:59.998426 8590 tablet.cc:2369] T ff9252832b7f45cc9076bca9a7ae8e5c P 83b39ca9a619489abe2c938e6047a01a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:36:00.005092 8760 raft_consensus.cc:482] T ff9252832b7f45cc9076bca9a7ae8e5c P 8b2ea59d9ae340fc8cee71292815b734 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:36:00.005239 8760 raft_consensus.cc:504] T ff9252832b7f45cc9076bca9a7ae8e5c P 8b2ea59d9ae340fc8cee71292815b734 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c088345e1e7f4a43a101bc3ebe83172a" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 46377 } } peers { permanent_uuid: "83b39ca9a619489abe2c938e6047a01a" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 34205 } } peers { permanent_uuid: "8b2ea59d9ae340fc8cee71292815b734" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 34359 } }
I20260925 22:36:00.005653 8760 leader_election.cc:288] T ff9252832b7f45cc9076bca9a7ae8e5c P 8b2ea59d9ae340fc8cee71292815b734 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c088345e1e7f4a43a101bc3ebe83172a (127.6.246.193:46377), 83b39ca9a619489abe2c938e6047a01a (127.6.246.194:34205)
I20260925 22:36:00.006165 8403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9252832b7f45cc9076bca9a7ae8e5c" candidate_uuid: "8b2ea59d9ae340fc8cee71292815b734" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c088345e1e7f4a43a101bc3ebe83172a" is_pre_election: true
I20260925 22:36:00.006315 8403 raft_consensus.cc:2454] T ff9252832b7f45cc9076bca9a7ae8e5c P c088345e1e7f4a43a101bc3ebe83172a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b2ea59d9ae340fc8cee71292815b734 in term 0.
I20260925 22:36:00.006247 8543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9252832b7f45cc9076bca9a7ae8e5c" candidate_uuid: "8b2ea59d9ae340fc8cee71292815b734" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b39ca9a619489abe2c938e6047a01a" is_pre_election: true
I20260925 22:36:00.006471 8543 raft_consensus.cc:2454] T ff9252832b7f45cc9076bca9a7ae8e5c P 83b39ca9a619489abe2c938e6047a01a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b2ea59d9ae340fc8cee71292815b734 in term 0.
I20260925 22:36:00.006626 8617 leader_election.cc:302] T ff9252832b7f45cc9076bca9a7ae8e5c P 8b2ea59d9ae340fc8cee71292815b734 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b2ea59d9ae340fc8cee71292815b734, c088345e1e7f4a43a101bc3ebe83172a; no voters:
I20260925 22:36:00.006800 8760 raft_consensus.cc:2781] T ff9252832b7f45cc9076bca9a7ae8e5c P 8b2ea59d9ae340fc8cee71292815b734 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:36:00.006861 8760 raft_consensus.cc:482] T ff9252832b7f45cc9076bca9a7ae8e5c P 8b2ea59d9ae340fc8cee71292815b734 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:36:00.006902 8760 raft_consensus.cc:3037] T ff9252832b7f45cc9076bca9a7ae8e5c P 8b2ea59d9ae340fc8cee71292815b734 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:36:00.007861 8760 raft_consensus.cc:504] T ff9252832b7f45cc9076bca9a7ae8e5c P 8b2ea59d9ae340fc8cee71292815b734 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c088345e1e7f4a43a101bc3ebe83172a" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 46377 } } peers { permanent_uuid: "83b39ca9a619489abe2c938e6047a01a" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 34205 } } peers { permanent_uuid: "8b2ea59d9ae340fc8cee71292815b734" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 34359 } }
I20260925 22:36:00.008203 8760 leader_election.cc:288] T ff9252832b7f45cc9076bca9a7ae8e5c P 8b2ea59d9ae340fc8cee71292815b734 [CANDIDATE]: Term 1 election: Requested vote from peers c088345e1e7f4a43a101bc3ebe83172a (127.6.246.193:46377), 83b39ca9a619489abe2c938e6047a01a (127.6.246.194:34205)
I20260925 22:36:00.008602 8543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9252832b7f45cc9076bca9a7ae8e5c" candidate_uuid: "8b2ea59d9ae340fc8cee71292815b734" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b39ca9a619489abe2c938e6047a01a"
I20260925 22:36:00.008603 8403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9252832b7f45cc9076bca9a7ae8e5c" candidate_uuid: "8b2ea59d9ae340fc8cee71292815b734" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c088345e1e7f4a43a101bc3ebe83172a"
I20260925 22:36:00.008734 8543 raft_consensus.cc:3037] T ff9252832b7f45cc9076bca9a7ae8e5c P 83b39ca9a619489abe2c938e6047a01a [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:36:00.008744 8403 raft_consensus.cc:3037] T ff9252832b7f45cc9076bca9a7ae8e5c P c088345e1e7f4a43a101bc3ebe83172a [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:36:00.009646 8403 raft_consensus.cc:2454] T ff9252832b7f45cc9076bca9a7ae8e5c P c088345e1e7f4a43a101bc3ebe83172a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b2ea59d9ae340fc8cee71292815b734 in term 1.
I20260925 22:36:00.009648 8543 raft_consensus.cc:2454] T ff9252832b7f45cc9076bca9a7ae8e5c P 83b39ca9a619489abe2c938e6047a01a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b2ea59d9ae340fc8cee71292815b734 in term 1.
I20260925 22:36:00.009927 8617 leader_election.cc:302] T ff9252832b7f45cc9076bca9a7ae8e5c P 8b2ea59d9ae340fc8cee71292815b734 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b2ea59d9ae340fc8cee71292815b734, c088345e1e7f4a43a101bc3ebe83172a; no voters:
I20260925 22:36:00.010094 8760 raft_consensus.cc:2781] T ff9252832b7f45cc9076bca9a7ae8e5c P 8b2ea59d9ae340fc8cee71292815b734 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:36:00.010179 8760 raft_consensus.cc:686] T ff9252832b7f45cc9076bca9a7ae8e5c P 8b2ea59d9ae340fc8cee71292815b734 [term 1 LEADER]: Becoming Leader. State: Replica: 8b2ea59d9ae340fc8cee71292815b734, State: Running, Role: LEADER
I20260925 22:36:00.010327 8760 consensus_queue.cc:237] T ff9252832b7f45cc9076bca9a7ae8e5c P 8b2ea59d9ae340fc8cee71292815b734 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c088345e1e7f4a43a101bc3ebe83172a" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 46377 } } peers { permanent_uuid: "83b39ca9a619489abe2c938e6047a01a" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 34205 } } peers { permanent_uuid: "8b2ea59d9ae340fc8cee71292815b734" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 34359 } }
I20260925 22:36:00.011852 8246 catalog_manager.cc:5495] T ff9252832b7f45cc9076bca9a7ae8e5c P 8b2ea59d9ae340fc8cee71292815b734 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b2ea59d9ae340fc8cee71292815b734 (127.6.246.195). New cstate: current_term: 1 leader_uuid: "8b2ea59d9ae340fc8cee71292815b734" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c088345e1e7f4a43a101bc3ebe83172a" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 46377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b39ca9a619489abe2c938e6047a01a" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 34205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b2ea59d9ae340fc8cee71292815b734" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 34359 } health_report { overall_health: HEALTHY } } }
Sep 25 22:36:00 dist-test-slave-wjdr krb5kdc[8201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375758, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.254@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
I20260925 22:36:00.082644 8786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:00.074196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:44481 (local address 127.0.0.1:43280)
0925 22:36:00.074522 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:00.074535 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:00.074637 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:00.074884 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:00.074887 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:00.075285 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:00.075376 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:00.075382 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:00.076112 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:00.076115 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:00.076795 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:00.076804 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:00.077185 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:00.077188 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:00.077248 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:00.077757 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:00.077769 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:00.079420 (+ 1651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:36:00.081143 (+ 1723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:00.081150 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:00.081156 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:00.081397 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:00.081629 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:00.081631 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:00.081632 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:00.081708 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:00.082106 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:36:00.082111 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:36:00.082227 (+ 116us) client_negotiation.cc:769] Sending connection context
0925 22:36:00.082317 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 22:36:00.082463 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":108,"threads_started":1}
I20260925 22:36:00.082661 8787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:00.074266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43280 (local address 127.6.246.254:44481)
0925 22:36:00.074544 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:00.074553 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:00.074591 (+ 38us) server_negotiation.cc:407] Connection header received
0925 22:36:00.074720 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:00.074722 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:00.074762 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:00.074841 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:00.075475 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:00.076010 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:00.076888 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:00.077091 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:00.077165 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:00.079564 (+ 2399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:00.079567 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:00.079569 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:00.079589 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:00.081047 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:00.081480 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:00.081483 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:00.081484 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:00.081554 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:00.081820 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:00.081824 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:00.081825 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:00.082012 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:00.082098 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:00.082380 (+ 282us) server_negotiation.cc:299] Negotiation successful
0925 22:36:00.082524 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":211,"thread_start_us":124,"threads_started":1}
I20260925 22:36:00.086490 8246 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.6.246.254@KRBTEST.COM'} at 127.0.0.1:43280:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 22:36:00.087230 8246 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:36:00.091619 8369 tablet_service.cc:1467] Processing CreateTablet for tablet c2eaf9292b1a47a19ded5113fe705406 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b8012dfe93384237b77a8eb1cf8b3ab2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 22:36:00.091693 8523 tablet_service.cc:1467] Processing CreateTablet for tablet c2eaf9292b1a47a19ded5113fe705406 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b8012dfe93384237b77a8eb1cf8b3ab2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 22:36:00.092007 8369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2eaf9292b1a47a19ded5113fe705406. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:36:00.092128 8523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2eaf9292b1a47a19ded5113fe705406. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:36:00.092212 8663 tablet_service.cc:1467] Processing CreateTablet for tablet c2eaf9292b1a47a19ded5113fe705406 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b8012dfe93384237b77a8eb1cf8b3ab2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 22:36:00.092581 8663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2eaf9292b1a47a19ded5113fe705406. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:36:00.094520 8754 tablet_bootstrap.cc:492] T c2eaf9292b1a47a19ded5113fe705406 P 83b39ca9a619489abe2c938e6047a01a: Bootstrap starting.
I20260925 22:36:00.094902 8752 tablet_bootstrap.cc:492] T c2eaf9292b1a47a19ded5113fe705406 P c088345e1e7f4a43a101bc3ebe83172a: Bootstrap starting.
I20260925 22:36:00.095537 8754 tablet_bootstrap.cc:654] T c2eaf9292b1a47a19ded5113fe705406 P 83b39ca9a619489abe2c938e6047a01a: Neither blocks nor log segments found. Creating new log.
I20260925 22:36:00.095541 8753 tablet_bootstrap.cc:492] T c2eaf9292b1a47a19ded5113fe705406 P 8b2ea59d9ae340fc8cee71292815b734: Bootstrap starting.
I20260925 22:36:00.095777 8752 tablet_bootstrap.cc:654] T c2eaf9292b1a47a19ded5113fe705406 P c088345e1e7f4a43a101bc3ebe83172a: Neither blocks nor log segments found. Creating new log.
I20260925 22:36:00.096444 8753 tablet_bootstrap.cc:654] T c2eaf9292b1a47a19ded5113fe705406 P 8b2ea59d9ae340fc8cee71292815b734: Neither blocks nor log segments found. Creating new log.
I20260925 22:36:00.096822 8754 tablet_bootstrap.cc:492] T c2eaf9292b1a47a19ded5113fe705406 P 83b39ca9a619489abe2c938e6047a01a: No bootstrap required, opened a new log
I20260925 22:36:00.096925 8754 ts_tablet_manager.cc:1397] T c2eaf9292b1a47a19ded5113fe705406 P 83b39ca9a619489abe2c938e6047a01a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260925 22:36:00.097389 8754 raft_consensus.cc:348] T c2eaf9292b1a47a19ded5113fe705406 P 83b39ca9a619489abe2c938e6047a01a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c088345e1e7f4a43a101bc3ebe83172a" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 46377 } } peers { permanent_uuid: "83b39ca9a619489abe2c938e6047a01a" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 34205 } } peers { permanent_uuid: "8b2ea59d9ae340fc8cee71292815b734" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 34359 } }
I20260925 22:36:00.097527 8754 raft_consensus.cc:374] T c2eaf9292b1a47a19ded5113fe705406 P 83b39ca9a619489abe2c938e6047a01a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:36:00.097563 8754 raft_consensus.cc:729] T c2eaf9292b1a47a19ded5113fe705406 P 83b39ca9a619489abe2c938e6047a01a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b39ca9a619489abe2c938e6047a01a, State: Initialized, Role: FOLLOWER
I20260925 22:36:00.097707 8754 consensus_queue.cc:260] T c2eaf9292b1a47a19ded5113fe705406 P 83b39ca9a619489abe2c938e6047a01a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c088345e1e7f4a43a101bc3ebe83172a" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 46377 } } peers { permanent_uuid: "83b39ca9a619489abe2c938e6047a01a" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 34205 } } peers { permanent_uuid: "8b2ea59d9ae340fc8cee71292815b734" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 34359 } }
I20260925 22:36:00.097831 8753 tablet_bootstrap.cc:492] T c2eaf9292b1a47a19ded5113fe705406 P 8b2ea59d9ae340fc8cee71292815b734: No bootstrap required, opened a new log
I20260925 22:36:00.097909 8753 ts_tablet_manager.cc:1397] T c2eaf9292b1a47a19ded5113fe705406 P 8b2ea59d9ae340fc8cee71292815b734: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 22:36:00.097950 8758 tablet_replica.cc:439] T c2eaf9292b1a47a19ded5113fe705406 P 83b39ca9a619489abe2c938e6047a01a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c088345e1e7f4a43a101bc3ebe83172a" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 46377 } } peers { permanent_uuid: "83b39ca9a619489abe2c938e6047a01a" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 34205 } } peers { permanent_uuid: "8b2ea59d9ae340fc8cee71292815b734" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 34359 } } }
I20260925 22:36:00.098094 8758 tablet_replica.cc:442] T c2eaf9292b1a47a19ded5113fe705406 P 83b39ca9a619489abe2c938e6047a01a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:36:00.098294 8753 raft_consensus.cc:348] T c2eaf9292b1a47a19ded5113fe705406 P 8b2ea59d9ae340fc8cee71292815b734 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c088345e1e7f4a43a101bc3ebe83172a" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 46377 } } peers { permanent_uuid: "83b39ca9a619489abe2c938e6047a01a" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 34205 } } peers { permanent_uuid: "8b2ea59d9ae340fc8cee71292815b734" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 34359 } }
I20260925 22:36:00.098400 8753 raft_consensus.cc:374] T c2eaf9292b1a47a19ded5113fe705406 P 8b2ea59d9ae340fc8cee71292815b734 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:36:00.098436 8752 tablet_bootstrap.cc:492] T c2eaf9292b1a47a19ded5113fe705406 P c088345e1e7f4a43a101bc3ebe83172a: No bootstrap required, opened a new log
I20260925 22:36:00.098433 8753 raft_consensus.cc:729] T c2eaf9292b1a47a19ded5113fe705406 P 8b2ea59d9ae340fc8cee71292815b734 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b2ea59d9ae340fc8cee71292815b734, State: Initialized, Role: FOLLOWER
I20260925 22:36:00.098524 8752 ts_tablet_manager.cc:1397] T c2eaf9292b1a47a19ded5113fe705406 P c088345e1e7f4a43a101bc3ebe83172a: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260925 22:36:00.098562 8753 consensus_queue.cc:260] T c2eaf9292b1a47a19ded5113fe705406 P 8b2ea59d9ae340fc8cee71292815b734 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c088345e1e7f4a43a101bc3ebe83172a" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 46377 } } peers { permanent_uuid: "83b39ca9a619489abe2c938e6047a01a" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 34205 } } peers { permanent_uuid: "8b2ea59d9ae340fc8cee71292815b734" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 34359 } }
I20260925 22:36:00.098820 8753 ts_tablet_manager.cc:1428] T c2eaf9292b1a47a19ded5113fe705406 P 8b2ea59d9ae340fc8cee71292815b734: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:36:00.098775 8760 tablet_replica.cc:439] T c2eaf9292b1a47a19ded5113fe705406 P 8b2ea59d9ae340fc8cee71292815b734: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c088345e1e7f4a43a101bc3ebe83172a" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 46377 } } peers { permanent_uuid: "83b39ca9a619489abe2c938e6047a01a" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 34205 } } peers { permanent_uuid: "8b2ea59d9ae340fc8cee71292815b734" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 34359 } } }
I20260925 22:36:00.098903 8760 tablet_replica.cc:442] T c2eaf9292b1a47a19ded5113fe705406 P 8b2ea59d9ae340fc8cee71292815b734: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:36:00.098935 8752 raft_consensus.cc:348] T c2eaf9292b1a47a19ded5113fe705406 P c088345e1e7f4a43a101bc3ebe83172a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c088345e1e7f4a43a101bc3ebe83172a" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 46377 } } peers { permanent_uuid: "83b39ca9a619489abe2c938e6047a01a" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 34205 } } peers { permanent_uuid: "8b2ea59d9ae340fc8cee71292815b734" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 34359 } }
I20260925 22:36:00.099026 8752 raft_consensus.cc:374] T c2eaf9292b1a47a19ded5113fe705406 P c088345e1e7f4a43a101bc3ebe83172a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:36:00.099061 8752 raft_consensus.cc:729] T c2eaf9292b1a47a19ded5113fe705406 P c088345e1e7f4a43a101bc3ebe83172a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c088345e1e7f4a43a101bc3ebe83172a, State: Initialized, Role: FOLLOWER
I20260925 22:36:00.099188 8752 consensus_queue.cc:260] T c2eaf9292b1a47a19ded5113fe705406 P c088345e1e7f4a43a101bc3ebe83172a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c088345e1e7f4a43a101bc3ebe83172a" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 46377 } } peers { permanent_uuid: "83b39ca9a619489abe2c938e6047a01a" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 34205 } } peers { permanent_uuid: "8b2ea59d9ae340fc8cee71292815b734" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 34359 } }
I20260925 22:36:00.099480 8752 ts_tablet_manager.cc:1428] T c2eaf9292b1a47a19ded5113fe705406 P c088345e1e7f4a43a101bc3ebe83172a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260925 22:36:00.099689 8759 tablet_replica.cc:439] T c2eaf9292b1a47a19ded5113fe705406 P c088345e1e7f4a43a101bc3ebe83172a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c088345e1e7f4a43a101bc3ebe83172a" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 46377 } } peers { permanent_uuid: "83b39ca9a619489abe2c938e6047a01a" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 34205 } } peers { permanent_uuid: "8b2ea59d9ae340fc8cee71292815b734" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 34359 } } }
I20260925 22:36:00.099821 8759 tablet_replica.cc:442] T c2eaf9292b1a47a19ded5113fe705406 P c088345e1e7f4a43a101bc3ebe83172a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:36:00.097950 8754 ts_tablet_manager.cc:1428] T c2eaf9292b1a47a19ded5113fe705406 P 83b39ca9a619489abe2c938e6047a01a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:36:00.211210 8759 raft_consensus.cc:482] T c2eaf9292b1a47a19ded5113fe705406 P c088345e1e7f4a43a101bc3ebe83172a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:36:00.211345 8759 raft_consensus.cc:504] T c2eaf9292b1a47a19ded5113fe705406 P c088345e1e7f4a43a101bc3ebe83172a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c088345e1e7f4a43a101bc3ebe83172a" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 46377 } } peers { permanent_uuid: "83b39ca9a619489abe2c938e6047a01a" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 34205 } } peers { permanent_uuid: "8b2ea59d9ae340fc8cee71292815b734" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 34359 } }
I20260925 22:36:00.212502 8759 leader_election.cc:288] T c2eaf9292b1a47a19ded5113fe705406 P c088345e1e7f4a43a101bc3ebe83172a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83b39ca9a619489abe2c938e6047a01a (127.6.246.194:34205), 8b2ea59d9ae340fc8cee71292815b734 (127.6.246.195:34359)
I20260925 22:36:00.217803 8750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:00.212891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.193:39405 (local address 127.6.246.195:34359)
0925 22:36:00.213040 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:00.213044 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:00.213342 (+ 298us) server_negotiation.cc:407] Connection header received
0925 22:36:00.213473 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:00.213476 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:00.213524 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:00.213604 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:36:00.213919 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:00.214445 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:00.216418 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:00.217056 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:00.217174 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:00.217227 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:00.217601 (+ 374us) server_negotiation.cc:299] Negotiation successful
0925 22:36:00.217673 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 22:36:00.217823 8793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:00.212844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.193:33615 (local address 127.6.246.194:34205)
0925 22:36:00.213133 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:00.213139 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:00.213157 (+ 18us) server_negotiation.cc:407] Connection header received
0925 22:36:00.213243 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:00.213249 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:00.213322 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:00.213424 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:36:00.213710 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:00.214459 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:00.216425 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:00.217046 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:00.217174 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:00.217227 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:00.217601 (+ 374us) server_negotiation.cc:299] Negotiation successful
0925 22:36:00.217673 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":90,"threads_started":1}
I20260925 22:36:00.217902 8791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:00.212687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.194:34205 (local address 127.6.246.193:33615)
0925 22:36:00.213041 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:00.213057 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:00.213156 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:00.213457 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:00.213460 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:00.213477 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:36:00.213600 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:00.213607 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:00.214576 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:00.214578 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:00.216250 (+ 1672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:00.216260 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:00.217198 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:00.217203 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:00.217383 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:00.217486 (+ 103us) client_negotiation.cc:769] Sending connection context
0925 22:36:00.217572 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 22:36:00.217713 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"mutex_wait_us":247,"thread_start_us":106,"threads_started":1}
I20260925 22:36:00.217912 8792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:00.212795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.195:34359 (local address 127.6.246.193:39405)
0925 22:36:00.213208 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:00.213232 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:00.213365 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:00.213624 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:00.213626 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:00.213641 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:36:00.213807 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:00.213814 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:00.214564 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:00.214568 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:00.216250 (+ 1682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:00.216260 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:00.217198 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:00.217203 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:00.217378 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:00.217484 (+ 106us) client_negotiation.cc:769] Sending connection context
0925 22:36:00.217572 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 22:36:00.217711 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"mutex_wait_us":389,"thread_start_us":110,"threads_started":1}
I20260925 22:36:00.218472 8543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2eaf9292b1a47a19ded5113fe705406" candidate_uuid: "c088345e1e7f4a43a101bc3ebe83172a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b39ca9a619489abe2c938e6047a01a" is_pre_election: true
I20260925 22:36:00.218627 8543 raft_consensus.cc:2454] T c2eaf9292b1a47a19ded5113fe705406 P 83b39ca9a619489abe2c938e6047a01a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c088345e1e7f4a43a101bc3ebe83172a in term 0.
I20260925 22:36:00.218719 8683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2eaf9292b1a47a19ded5113fe705406" candidate_uuid: "c088345e1e7f4a43a101bc3ebe83172a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b2ea59d9ae340fc8cee71292815b734" is_pre_election: true
I20260925 22:36:00.218962 8683 raft_consensus.cc:2454] T c2eaf9292b1a47a19ded5113fe705406 P 8b2ea59d9ae340fc8cee71292815b734 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c088345e1e7f4a43a101bc3ebe83172a in term 0.
I20260925 22:36:00.219106 8337 leader_election.cc:302] T c2eaf9292b1a47a19ded5113fe705406 P c088345e1e7f4a43a101bc3ebe83172a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b39ca9a619489abe2c938e6047a01a, c088345e1e7f4a43a101bc3ebe83172a; no voters:
I20260925 22:36:00.219398 8759 raft_consensus.cc:2781] T c2eaf9292b1a47a19ded5113fe705406 P c088345e1e7f4a43a101bc3ebe83172a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:36:00.219455 8759 raft_consensus.cc:482] T c2eaf9292b1a47a19ded5113fe705406 P c088345e1e7f4a43a101bc3ebe83172a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:36:00.219491 8759 raft_consensus.cc:3037] T c2eaf9292b1a47a19ded5113fe705406 P c088345e1e7f4a43a101bc3ebe83172a [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:36:00.220635 8759 raft_consensus.cc:504] T c2eaf9292b1a47a19ded5113fe705406 P c088345e1e7f4a43a101bc3ebe83172a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c088345e1e7f4a43a101bc3ebe83172a" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 46377 } } peers { permanent_uuid: "83b39ca9a619489abe2c938e6047a01a" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 34205 } } peers { permanent_uuid: "8b2ea59d9ae340fc8cee71292815b734" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 34359 } }
I20260925 22:36:00.220964 8759 leader_election.cc:288] T c2eaf9292b1a47a19ded5113fe705406 P c088345e1e7f4a43a101bc3ebe83172a [CANDIDATE]: Term 1 election: Requested vote from peers 83b39ca9a619489abe2c938e6047a01a (127.6.246.194:34205), 8b2ea59d9ae340fc8cee71292815b734 (127.6.246.195:34359)
I20260925 22:36:00.221280 8683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2eaf9292b1a47a19ded5113fe705406" candidate_uuid: "c088345e1e7f4a43a101bc3ebe83172a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b2ea59d9ae340fc8cee71292815b734"
I20260925 22:36:00.221287 8543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2eaf9292b1a47a19ded5113fe705406" candidate_uuid: "c088345e1e7f4a43a101bc3ebe83172a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b39ca9a619489abe2c938e6047a01a"
I20260925 22:36:00.221377 8683 raft_consensus.cc:3037] T c2eaf9292b1a47a19ded5113fe705406 P 8b2ea59d9ae340fc8cee71292815b734 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:36:00.221398 8543 raft_consensus.cc:3037] T c2eaf9292b1a47a19ded5113fe705406 P 83b39ca9a619489abe2c938e6047a01a [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:36:00.222294 8543 raft_consensus.cc:2454] T c2eaf9292b1a47a19ded5113fe705406 P 83b39ca9a619489abe2c938e6047a01a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c088345e1e7f4a43a101bc3ebe83172a in term 1.
I20260925 22:36:00.222368 8683 raft_consensus.cc:2454] T c2eaf9292b1a47a19ded5113fe705406 P 8b2ea59d9ae340fc8cee71292815b734 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c088345e1e7f4a43a101bc3ebe83172a in term 1.
I20260925 22:36:00.222561 8337 leader_election.cc:302] T c2eaf9292b1a47a19ded5113fe705406 P c088345e1e7f4a43a101bc3ebe83172a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b39ca9a619489abe2c938e6047a01a, c088345e1e7f4a43a101bc3ebe83172a; no voters:
I20260925 22:36:00.222723 8759 raft_consensus.cc:2781] T c2eaf9292b1a47a19ded5113fe705406 P c088345e1e7f4a43a101bc3ebe83172a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:36:00.223138 8759 raft_consensus.cc:686] T c2eaf9292b1a47a19ded5113fe705406 P c088345e1e7f4a43a101bc3ebe83172a [term 1 LEADER]: Becoming Leader. State: Replica: c088345e1e7f4a43a101bc3ebe83172a, State: Running, Role: LEADER
I20260925 22:36:00.223409 8759 consensus_queue.cc:237] T c2eaf9292b1a47a19ded5113fe705406 P c088345e1e7f4a43a101bc3ebe83172a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c088345e1e7f4a43a101bc3ebe83172a" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 46377 } } peers { permanent_uuid: "83b39ca9a619489abe2c938e6047a01a" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 34205 } } peers { permanent_uuid: "8b2ea59d9ae340fc8cee71292815b734" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 34359 } }
I20260925 22:36:00.224565 8794 tablet_replica.cc:439] T c2eaf9292b1a47a19ded5113fe705406 P c088345e1e7f4a43a101bc3ebe83172a: TxnStatusTablet state changed. Reason: New leader c088345e1e7f4a43a101bc3ebe83172a. Latest consensus state: current_term: 1 leader_uuid: "c088345e1e7f4a43a101bc3ebe83172a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c088345e1e7f4a43a101bc3ebe83172a" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 46377 } } peers { permanent_uuid: "83b39ca9a619489abe2c938e6047a01a" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 34205 } } peers { permanent_uuid: "8b2ea59d9ae340fc8cee71292815b734" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 34359 } } }
I20260925 22:36:00.224668 8794 tablet_replica.cc:442] T c2eaf9292b1a47a19ded5113fe705406 P c088345e1e7f4a43a101bc3ebe83172a: This TxnStatusTablet replica's current role is: LEADER
I20260925 22:36:00.224943 8796 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 22:36:00.225092 8796 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 22:36:00.225716 8246 catalog_manager.cc:5495] T c2eaf9292b1a47a19ded5113fe705406 P c088345e1e7f4a43a101bc3ebe83172a reported cstate change: term changed from 0 to 1, leader changed from <none> to c088345e1e7f4a43a101bc3ebe83172a (127.6.246.193). New cstate: current_term: 1 leader_uuid: "c088345e1e7f4a43a101bc3ebe83172a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c088345e1e7f4a43a101bc3ebe83172a" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 46377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b39ca9a619489abe2c938e6047a01a" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 34205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b2ea59d9ae340fc8cee71292815b734" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 34359 } health_report { overall_health: UNKNOWN } } }
I20260925 22:36:00.253793 8786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:00.249496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.193:46377 (local address 127.0.0.1:56478)
0925 22:36:00.249738 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:00.249754 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:00.249868 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:00.250200 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:00.250203 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:00.250211 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 22:36:00.250318 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:00.250324 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:00.251255 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:00.251261 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:00.252124 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:00.252133 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:00.252710 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:00.252713 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:00.252810 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:00.252859 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 22:36:00.253485 (+ 626us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 22:36:00.253501 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 22:36:00.253571 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 22:36:00.253656 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141}
I20260925 22:36:00.253804 8799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:00.249551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56478 (local address 127.6.246.193:46377)
0925 22:36:00.249785 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:00.249792 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:00.249808 (+ 16us) server_negotiation.cc:407] Connection header received
0925 22:36:00.250013 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:00.250016 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:00.250076 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:00.250189 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:36:00.250431 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:00.251151 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:00.252244 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:00.252608 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:00.252712 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:00.253017 (+ 305us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:36:00.253388 (+ 371us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:36:00.253509 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:00.253581 (+ 72us) server_negotiation.cc:299] Negotiation successful
0925 22:36:00.253629 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":53,"threads_started":1}
I20260925 22:36:00.256618 8543 raft_consensus.cc:1261] T c2eaf9292b1a47a19ded5113fe705406 P 83b39ca9a619489abe2c938e6047a01a [term 1 FOLLOWER]: Refusing update from remote peer c088345e1e7f4a43a101bc3ebe83172a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:36:00.256675 8683 raft_consensus.cc:1261] T c2eaf9292b1a47a19ded5113fe705406 P 8b2ea59d9ae340fc8cee71292815b734 [term 1 FOLLOWER]: Refusing update from remote peer c088345e1e7f4a43a101bc3ebe83172a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:36:00.257200 8794 consensus_queue.cc:1035] T c2eaf9292b1a47a19ded5113fe705406 P c088345e1e7f4a43a101bc3ebe83172a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b2ea59d9ae340fc8cee71292815b734" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 34359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:36:00.257663 8794 consensus_queue.cc:1035] T c2eaf9292b1a47a19ded5113fe705406 P c088345e1e7f4a43a101bc3ebe83172a [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b39ca9a619489abe2c938e6047a01a" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 34205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:36:00.258843 8760 tablet_replica.cc:439] T c2eaf9292b1a47a19ded5113fe705406 P 8b2ea59d9ae340fc8cee71292815b734: TxnStatusTablet state changed. Reason: New leader c088345e1e7f4a43a101bc3ebe83172a. Latest consensus state: current_term: 1 leader_uuid: "c088345e1e7f4a43a101bc3ebe83172a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c088345e1e7f4a43a101bc3ebe83172a" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 46377 } } peers { permanent_uuid: "83b39ca9a619489abe2c938e6047a01a" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 34205 } } peers { permanent_uuid: "8b2ea59d9ae340fc8cee71292815b734" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 34359 } } }
I20260925 22:36:00.258952 8760 tablet_replica.cc:442] T c2eaf9292b1a47a19ded5113fe705406 P 8b2ea59d9ae340fc8cee71292815b734: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:36:00.258960 8758 tablet_replica.cc:439] T c2eaf9292b1a47a19ded5113fe705406 P 83b39ca9a619489abe2c938e6047a01a: TxnStatusTablet state changed. Reason: New leader c088345e1e7f4a43a101bc3ebe83172a. Latest consensus state: current_term: 1 leader_uuid: "c088345e1e7f4a43a101bc3ebe83172a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c088345e1e7f4a43a101bc3ebe83172a" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 46377 } } peers { permanent_uuid: "83b39ca9a619489abe2c938e6047a01a" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 34205 } } peers { permanent_uuid: "8b2ea59d9ae340fc8cee71292815b734" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 34359 } } }
I20260925 22:36:00.259056 8758 tablet_replica.cc:442] T c2eaf9292b1a47a19ded5113fe705406 P 83b39ca9a619489abe2c938e6047a01a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:36:00.265785 8759 tablet_replica.cc:439] T c2eaf9292b1a47a19ded5113fe705406 P c088345e1e7f4a43a101bc3ebe83172a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c088345e1e7f4a43a101bc3ebe83172a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c088345e1e7f4a43a101bc3ebe83172a" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 46377 } } peers { permanent_uuid: "83b39ca9a619489abe2c938e6047a01a" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 34205 } } peers { permanent_uuid: "8b2ea59d9ae340fc8cee71292815b734" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 34359 } } }
I20260925 22:36:00.265903 8759 tablet_replica.cc:442] T c2eaf9292b1a47a19ded5113fe705406 P c088345e1e7f4a43a101bc3ebe83172a: This TxnStatusTablet replica's current role is: LEADER
I20260925 22:36:00.265918 8794 tablet_replica.cc:439] T c2eaf9292b1a47a19ded5113fe705406 P c088345e1e7f4a43a101bc3ebe83172a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c088345e1e7f4a43a101bc3ebe83172a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c088345e1e7f4a43a101bc3ebe83172a" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 46377 } } peers { permanent_uuid: "83b39ca9a619489abe2c938e6047a01a" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 34205 } } peers { permanent_uuid: "8b2ea59d9ae340fc8cee71292815b734" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 34359 } } }
I20260925 22:36:00.266000 8794 tablet_replica.cc:442] T c2eaf9292b1a47a19ded5113fe705406 P c088345e1e7f4a43a101bc3ebe83172a: This TxnStatusTablet replica's current role is: LEADER
I20260925 22:36:00.266465 8760 tablet_replica.cc:439] T c2eaf9292b1a47a19ded5113fe705406 P 8b2ea59d9ae340fc8cee71292815b734: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c088345e1e7f4a43a101bc3ebe83172a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c088345e1e7f4a43a101bc3ebe83172a" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 46377 } } peers { permanent_uuid: "83b39ca9a619489abe2c938e6047a01a" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 34205 } } peers { permanent_uuid: "8b2ea59d9ae340fc8cee71292815b734" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 34359 } } }
I20260925 22:36:00.266564 8760 tablet_replica.cc:442] T c2eaf9292b1a47a19ded5113fe705406 P 8b2ea59d9ae340fc8cee71292815b734: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:36:00.268658 8758 tablet_replica.cc:439] T c2eaf9292b1a47a19ded5113fe705406 P 83b39ca9a619489abe2c938e6047a01a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c088345e1e7f4a43a101bc3ebe83172a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c088345e1e7f4a43a101bc3ebe83172a" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 46377 } } peers { permanent_uuid: "83b39ca9a619489abe2c938e6047a01a" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 34205 } } peers { permanent_uuid: "8b2ea59d9ae340fc8cee71292815b734" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 34359 } } }
I20260925 22:36:00.268757 8758 tablet_replica.cc:442] T c2eaf9292b1a47a19ded5113fe705406 P 83b39ca9a619489abe2c938e6047a01a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:36:00.280710 8799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:00.277206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.195:52671 (local address 127.6.246.193:46377)
0925 22:36:00.277333 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:00.277342 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:00.277422 (+ 80us) server_negotiation.cc:407] Connection header received
0925 22:36:00.277542 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:00.277544 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:00.277582 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:00.277643 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:36:00.277865 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:00.278411 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:00.279640 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:00.279882 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:00.279971 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:00.280226 (+ 255us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:36:00.280284 (+ 58us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:36:00.280335 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:00.280446 (+ 111us) server_negotiation.cc:299] Negotiation successful
0925 22:36:00.280545 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 22:36:00.280707 8811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:00.277078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.193:46377 (local address 127.6.246.195:52671)
0925 22:36:00.277340 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:00.277355 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:00.277458 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:00.277667 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:00.277671 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:00.277678 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 22:36:00.277778 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:00.277784 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:00.278519 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:00.278523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:00.279537 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:00.279548 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:00.279978 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:00.279982 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:00.280066 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:00.280112 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 22:36:00.280359 (+ 247us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 22:36:00.280365 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 22:36:00.280436 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 22:36:00.280557 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":80,"threads_started":1}
I20260925 22:36:00.287191 8543 raft_consensus.cc:1261] T ff9252832b7f45cc9076bca9a7ae8e5c P 83b39ca9a619489abe2c938e6047a01a [term 1 FOLLOWER]: Refusing update from remote peer 8b2ea59d9ae340fc8cee71292815b734: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:36:00.287254 8403 raft_consensus.cc:1261] T ff9252832b7f45cc9076bca9a7ae8e5c P c088345e1e7f4a43a101bc3ebe83172a [term 1 FOLLOWER]: Refusing update from remote peer 8b2ea59d9ae340fc8cee71292815b734: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:36:00.287663 8762 consensus_queue.cc:1035] T ff9252832b7f45cc9076bca9a7ae8e5c P 8b2ea59d9ae340fc8cee71292815b734 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b39ca9a619489abe2c938e6047a01a" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 34205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:36:00.287811 8760 consensus_queue.cc:1035] T ff9252832b7f45cc9076bca9a7ae8e5c P 8b2ea59d9ae340fc8cee71292815b734 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c088345e1e7f4a43a101bc3ebe83172a" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 46377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:36:00.310842 8750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:00.306701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.193:33913 (local address 127.6.246.195:34359)
0925 22:36:00.306809 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:00.306814 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:00.307004 (+ 190us) server_negotiation.cc:407] Connection header received
0925 22:36:00.307140 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:00.307143 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:00.307190 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:00.307264 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:36:00.307611 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:00.308101 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:00.309385 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:00.309725 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:00.309795 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:00.310211 (+ 416us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:36:00.310306 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:36:00.310348 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:00.310700 (+ 352us) server_negotiation.cc:299] Negotiation successful
0925 22:36:00.310745 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 22:36:00.310923 8815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:00.306378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.195:34359 (local address 127.6.246.193:33913)
0925 22:36:00.306916 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:00.306928 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:00.307051 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:00.307367 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:00.307370 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:00.307379 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 22:36:00.307507 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:00.307513 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:00.308202 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:00.308205 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:00.309247 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:00.309259 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:00.309922 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:00.309926 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:00.310024 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:00.310070 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 22:36:00.310604 (+ 534us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 22:36:00.310610 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 22:36:00.310676 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 22:36:00.310753 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":107,"threads_started":1}
I20260925 22:36:00.332245 8246 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:36:00.332410 8246 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:36:00.334303 8246 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a097ad9e7471412b897b9d2feefb6d8a: Sending DeleteTablet for 3 replicas of tablet ff9252832b7f45cc9076bca9a7ae8e5c
I20260925 22:36:00.334877 8523 tablet_service.cc:1514] Processing DeleteTablet for tablet ff9252832b7f45cc9076bca9a7ae8e5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:36:00 UTC) from {username='kudu', principal='kudu/127.6.246.254@KRBTEST.COM'} at 127.0.0.1:37146
I20260925 22:36:00.334956 8663 tablet_service.cc:1514] Processing DeleteTablet for tablet ff9252832b7f45cc9076bca9a7ae8e5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:36:00 UTC) from {username='kudu', principal='kudu/127.6.246.254@KRBTEST.COM'} at 127.0.0.1:50422
I20260925 22:36:00.335161 8818 tablet_replica.cc:330] T ff9252832b7f45cc9076bca9a7ae8e5c P 83b39ca9a619489abe2c938e6047a01a: stopping tablet replica
I20260925 22:36:00.335238 8819 tablet_replica.cc:330] T ff9252832b7f45cc9076bca9a7ae8e5c P 8b2ea59d9ae340fc8cee71292815b734: stopping tablet replica
I20260925 22:36:00.335274 8818 raft_consensus.cc:2229] T ff9252832b7f45cc9076bca9a7ae8e5c P 83b39ca9a619489abe2c938e6047a01a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:36:00.335371 8818 raft_consensus.cc:2258] T ff9252832b7f45cc9076bca9a7ae8e5c P 83b39ca9a619489abe2c938e6047a01a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:36:00.335366 8819 raft_consensus.cc:2229] T ff9252832b7f45cc9076bca9a7ae8e5c P 8b2ea59d9ae340fc8cee71292815b734 [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:36:00.335527 8819 raft_consensus.cc:2258] T ff9252832b7f45cc9076bca9a7ae8e5c P 8b2ea59d9ae340fc8cee71292815b734 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:36:00.336110 8819 ts_tablet_manager.cc:1905] T ff9252832b7f45cc9076bca9a7ae8e5c P 8b2ea59d9ae340fc8cee71292815b734: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:36:00.336230 8818 ts_tablet_manager.cc:1905] T ff9252832b7f45cc9076bca9a7ae8e5c P 83b39ca9a619489abe2c938e6047a01a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:36:00.336292 8369 tablet_service.cc:1514] Processing DeleteTablet for tablet ff9252832b7f45cc9076bca9a7ae8e5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:36:00 UTC) from {username='kudu', principal='kudu/127.6.246.254@KRBTEST.COM'} at 127.0.0.1:56476
I20260925 22:36:00.336722 8821 tablet_replica.cc:330] T ff9252832b7f45cc9076bca9a7ae8e5c P c088345e1e7f4a43a101bc3ebe83172a: stopping tablet replica
I20260925 22:36:00.336910 8821 raft_consensus.cc:2229] T ff9252832b7f45cc9076bca9a7ae8e5c P c088345e1e7f4a43a101bc3ebe83172a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:36:00.337062 8821 raft_consensus.cc:2258] T ff9252832b7f45cc9076bca9a7ae8e5c P c088345e1e7f4a43a101bc3ebe83172a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:36:00.337946 8821 ts_tablet_manager.cc:1905] T ff9252832b7f45cc9076bca9a7ae8e5c P c088345e1e7f4a43a101bc3ebe83172a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:36:00.339237 8818 ts_tablet_manager.cc:1918] T ff9252832b7f45cc9076bca9a7ae8e5c P 83b39ca9a619489abe2c938e6047a01a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 22:36:00.339286 8819 ts_tablet_manager.cc:1918] T ff9252832b7f45cc9076bca9a7ae8e5c P 8b2ea59d9ae340fc8cee71292815b734: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 22:36:00.339319 8818 log.cc:1196] T ff9252832b7f45cc9076bca9a7ae8e5c P 83b39ca9a619489abe2c938e6047a01a: Deleting WAL directory at /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/minicluster-data/ts-1/wal/wals/ff9252832b7f45cc9076bca9a7ae8e5c
I20260925 22:36:00.339351 8819 log.cc:1196] T ff9252832b7f45cc9076bca9a7ae8e5c P 8b2ea59d9ae340fc8cee71292815b734: Deleting WAL directory at /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/minicluster-data/ts-2/wal/wals/ff9252832b7f45cc9076bca9a7ae8e5c
I20260925 22:36:00.339597 8819 ts_tablet_manager.cc:1939] T ff9252832b7f45cc9076bca9a7ae8e5c P 8b2ea59d9ae340fc8cee71292815b734: Deleting consensus metadata
I20260925 22:36:00.339630 8818 ts_tablet_manager.cc:1939] T ff9252832b7f45cc9076bca9a7ae8e5c P 83b39ca9a619489abe2c938e6047a01a: Deleting consensus metadata
I20260925 22:36:00.340358 8230 catalog_manager.cc:4841] TS 8b2ea59d9ae340fc8cee71292815b734 (127.6.246.195:34359): tablet ff9252832b7f45cc9076bca9a7ae8e5c (table test-table [id=a60ccddfef974f74ba005e9b88e79093]) successfully deleted
I20260925 22:36:00.340549 8233 catalog_manager.cc:4841] TS 83b39ca9a619489abe2c938e6047a01a (127.6.246.194:34205): tablet ff9252832b7f45cc9076bca9a7ae8e5c (table test-table [id=a60ccddfef974f74ba005e9b88e79093]) successfully deleted
I20260925 22:36:00.341543 8821 ts_tablet_manager.cc:1918] T ff9252832b7f45cc9076bca9a7ae8e5c P c088345e1e7f4a43a101bc3ebe83172a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 22:36:00.341624 8821 log.cc:1196] T ff9252832b7f45cc9076bca9a7ae8e5c P c088345e1e7f4a43a101bc3ebe83172a: Deleting WAL directory at /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790375752797504-7131-0/minicluster-data/ts-0/wal/wals/ff9252832b7f45cc9076bca9a7ae8e5c
I20260925 22:36:00.341871 8821 ts_tablet_manager.cc:1939] T ff9252832b7f45cc9076bca9a7ae8e5c P c088345e1e7f4a43a101bc3ebe83172a: Deleting consensus metadata
I20260925 22:36:00.342465 8233 catalog_manager.cc:4841] TS c088345e1e7f4a43a101bc3ebe83172a (127.6.246.193:46377): tablet ff9252832b7f45cc9076bca9a7ae8e5c (table test-table [id=a60ccddfef974f74ba005e9b88e79093]) successfully deleted
Sep 25 22:36:00 dist-test-slave-wjdr krb5kdc[8201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375759, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.246.193@KRBTEST.COM
I20260925 22:36:00.350488 8799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:00.337483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56486 (local address 127.6.246.193:46377)
0925 22:36:00.337581 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:00.337585 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:00.340401 (+ 2816us) server_negotiation.cc:407] Connection header received
0925 22:36:00.340667 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:00.340671 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:00.340716 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:00.340765 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:00.341771 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:00.342630 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:00.343382 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:00.343702 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:00.343810 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:00.346498 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:00.346505 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:00.346516 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:00.346536 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:00.348051 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:00.348546 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:00.348551 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:00.348552 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:00.348636 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:00.348947 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:00.348949 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:00.348950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:00.349569 (+ 619us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:00.349741 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:00.349990 (+ 249us) server_negotiation.cc:299] Negotiation successful
0925 22:36:00.350094 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
W20260925 22:36:00.350924 8442 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56486
I20260925 22:36:00.360129 8787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:00.352586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43284 (local address 127.6.246.254:44481)
0925 22:36:00.352726 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:00.352732 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:00.352843 (+ 111us) server_negotiation.cc:407] Connection header received
0925 22:36:00.352939 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:00.352943 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:00.352991 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:00.353063 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:00.353618 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:00.354455 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:00.355203 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:00.355602 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:00.355711 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:00.356960 (+ 1249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:00.356963 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:00.356966 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:00.356994 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:00.358517 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:00.358892 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:00.358895 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:00.358897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:00.358987 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:00.359266 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:00.359269 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:00.359271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:00.359484 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:00.359640 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:00.359834 (+ 194us) server_negotiation.cc:299] Negotiation successful
0925 22:36:00.359970 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260925 22:36:00.360543 8246 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43284
I20260925 22:36:00.361609 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 8318
I20260925 22:36:00.369264 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 8458
I20260925 22:36:00.374604 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 8598
I20260925 22:36:00.381184 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 8214
2026-09-25T22:36:00Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2982 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:36:00 dist-test-slave-wjdr krb5kdc[8827](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:36:00 dist-test-slave-wjdr krb5kdc[8827](info): set up 2 sockets
Sep 25 22:36:00 dist-test-slave-wjdr krb5kdc[8827](info): commencing operation
krb5kdc: starting...
W20260925 22:36:01.412330 7131 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Sep 25 22:36:01 dist-test-slave-wjdr krb5kdc[8827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:36:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:36:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.246.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab.
Entry for principal kudu/127.6.246.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab.
I20260925 22:36:01.531726 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:34691
--webserver_interface=127.6.246.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab
--principal=kudu/127.6.246.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:40793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.246.254:34691
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:36:01.638243 8840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:36:01.638525 8840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:36:01.638597 8840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:36:01.642519 8840 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:36:01.642599 8840 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 22:36:01.642632 8840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:36:01.642674 8840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:36:01.642712 8840 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:36:01.642750 8840 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:36:01.647815 8840 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:40793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.246.254:34691
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab
--principal=kudu/127.6.246.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:34691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.246.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8840
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:36:01.648831 8840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:36:01.649799 8840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:36:01.658442 8848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:01.658697 8846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:01.658650 8840 server_base.cc:1034] running on GCE node
W20260925 22:36:01.658906 8845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:01.659667 8840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:36:01.660866 8840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:36:01.662039 8840 hybrid_clock.cc:648] HybridClock initialized: now 1790375761662021 us; error 30 us; skew 500 ppm
Sep 25 22:36:01 dist-test-slave-wjdr krb5kdc[8827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375761, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:36:01.664618 8840 init.cc:377] Logged in from keytab as kudu/127.6.246.254@KRBTEST.COM (short username kudu)
I20260925 22:36:01.665504 8840 webserver.cc:460] Webserver started at http://127.6.246.254:44477/ using document root <none> and password file <none>
I20260925 22:36:01.666045 8840 fs_manager.cc:359] Metadata directory not provided
I20260925 22:36:01.666131 8840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:36:01.666329 8840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:36:01.668248 8840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/minicluster-data/master-0/data/instance:
uuid: "5f739135c3c04d60a053abf48962a868"
format_stamp: "Formatted at 2026-09-25 22:36:01 on dist-test-slave-wjdr"
server_key: "ddca9d3b492a223d5bda06fc5a03086a"
server_key_iv: "d4fbd3c6ecfafa30a69891a5bf05ba43"
server_key_version: "encryptionkey@0"
I20260925 22:36:01.668841 8840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/minicluster-data/master-0/wal/instance:
uuid: "5f739135c3c04d60a053abf48962a868"
format_stamp: "Formatted at 2026-09-25 22:36:01 on dist-test-slave-wjdr"
server_key: "ddca9d3b492a223d5bda06fc5a03086a"
server_key_iv: "d4fbd3c6ecfafa30a69891a5bf05ba43"
server_key_version: "encryptionkey@0"
I20260925 22:36:01.672492 8840 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:36:01.674726 8855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:01.675904 8840 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:36:01.676043 8840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/minicluster-data/master-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/minicluster-data/master-0/wal
uuid: "5f739135c3c04d60a053abf48962a868"
format_stamp: "Formatted at 2026-09-25 22:36:01 on dist-test-slave-wjdr"
server_key: "ddca9d3b492a223d5bda06fc5a03086a"
server_key_iv: "d4fbd3c6ecfafa30a69891a5bf05ba43"
server_key_version: "encryptionkey@0"
I20260925 22:36:01.676148 8840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:36:01.705440 8840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:36:01.708621 8840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:36:01.708837 8840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:36:01.716729 8840 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.254:34691
I20260925 22:36:01.716755 8917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.254:34691 every 8 connection(s)
I20260925 22:36:01.717687 8840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
I20260925 22:36:01.720381 8918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:36:01.725725 8918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f739135c3c04d60a053abf48962a868: Bootstrap starting.
I20260925 22:36:01.727252 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 8840
I20260925 22:36:01.727371 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/minicluster-data/master-0/wal/instance
I20260925 22:36:01.727627 7131 external_mini_cluster.cc:1428] Setting key f7e0b7116300081771f02cd670292240
I20260925 22:36:01.728447 8918 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f739135c3c04d60a053abf48962a868: Neither blocks nor log segments found. Creating new log.
I20260925 22:36:01.729378 8918 log.cc:824] T 00000000000000000000000000000000 P 5f739135c3c04d60a053abf48962a868: Log is configured to *not* fsync() on all Append() calls
I20260925 22:36:01.731529 8918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f739135c3c04d60a053abf48962a868: No bootstrap required, opened a new log
I20260925 22:36:01.735169 8918 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f739135c3c04d60a053abf48962a868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f739135c3c04d60a053abf48962a868" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 34691 } }
I20260925 22:36:01.735414 8918 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f739135c3c04d60a053abf48962a868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:36:01.735491 8918 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f739135c3c04d60a053abf48962a868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f739135c3c04d60a053abf48962a868, State: Initialized, Role: FOLLOWER
I20260925 22:36:01.735997 8918 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f739135c3c04d60a053abf48962a868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f739135c3c04d60a053abf48962a868" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 34691 } }
I20260925 22:36:01.736182 8918 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f739135c3c04d60a053abf48962a868 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:36:01.736248 8918 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f739135c3c04d60a053abf48962a868 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
Sep 25 22:36:01 dist-test-slave-wjdr krb5kdc[8827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
I20260925 22:36:01.736344 8918 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f739135c3c04d60a053abf48962a868 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:36:01.737296 8918 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f739135c3c04d60a053abf48962a868 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f739135c3c04d60a053abf48962a868" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 34691 } }
I20260925 22:36:01.737633 8918 leader_election.cc:302] T 00000000000000000000000000000000 P 5f739135c3c04d60a053abf48962a868 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f739135c3c04d60a053abf48962a868; no voters:
I20260925 22:36:01.737987 8918 leader_election.cc:288] T 00000000000000000000000000000000 P 5f739135c3c04d60a053abf48962a868 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:36:01.738073 8923 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f739135c3c04d60a053abf48962a868 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:36:01.738240 8923 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f739135c3c04d60a053abf48962a868 [term 1 LEADER]: Becoming Leader. State: Replica: 5f739135c3c04d60a053abf48962a868, State: Running, Role: LEADER
I20260925 22:36:01.738530 8923 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f739135c3c04d60a053abf48962a868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f739135c3c04d60a053abf48962a868" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 34691 } }
I20260925 22:36:01.739226 8918 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f739135c3c04d60a053abf48962a868 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:36:01.739665 8925 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f739135c3c04d60a053abf48962a868 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f739135c3c04d60a053abf48962a868. Latest consensus state: current_term: 1 leader_uuid: "5f739135c3c04d60a053abf48962a868" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f739135c3c04d60a053abf48962a868" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 34691 } } }
I20260925 22:36:01.739779 8925 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f739135c3c04d60a053abf48962a868 [sys.catalog]: This master's current role is: LEADER
I20260925 22:36:01.740069 8924 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f739135c3c04d60a053abf48962a868 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f739135c3c04d60a053abf48962a868" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f739135c3c04d60a053abf48962a868" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 34691 } } }
I20260925 22:36:01.740166 8924 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f739135c3c04d60a053abf48962a868 [sys.catalog]: This master's current role is: LEADER
I20260925 22:36:01.742617 8921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:01.728780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48466 (local address 127.6.246.254:34691)
0925 22:36:01.729209 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:01.729243 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:01.729277 (+ 34us) server_negotiation.cc:407] Connection header received
0925 22:36:01.730091 (+ 814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:01.730117 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:01.730442 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:01.730813 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:01.731598 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:01.732671 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:01.733521 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:01.734021 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:01.734193 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:01.736919 (+ 2726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:01.736925 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:01.736937 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:01.736962 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:01.738964 (+ 2002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:01.739421 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:01.739427 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:01.739432 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:01.739575 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:01.740024 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:01.740028 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:01.740029 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:01.740565 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:01.740758 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:01.741567 (+ 809us) server_negotiation.cc:299] Negotiation successful
0925 22:36:01.741859 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":128,"threads_started":1}
W20260925 22:36:01.744333 8938 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5f739135c3c04d60a053abf48962a868: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:36:01.744417 8938 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:36:01.744680 8927 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:36:01.745364 8927 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:36:01.750895 8927 catalog_manager.cc:1347] Generated new cluster ID: 2040faef42774ff9ad6c8fc88ebf6906
I20260925 22:36:01.750964 8927 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:36:01.780373 8927 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:36:01.781693 8927 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:36:01.792243 8927 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f739135c3c04d60a053abf48962a868: Generated new TSK 0
I20260925 22:36:01.793066 8927 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.246.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.193.keytab.
Entry for principal kudu/127.6.246.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.193.keytab.
I20260925 22:36:01.832001 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.193:0
--local_ip_for_outbound_sockets=127.6.246.193
--webserver_interface=127.6.246.193
--webserver_port=0
--tserver_master_addrs=127.6.246.254:34691
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.193.keytab
--principal=kudu/127.6.246.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:40793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:36:01.942502 8944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:36:01.942770 8944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:36:01.942816 8944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:36:01.946763 8944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:36:01.946827 8944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:36:01.946884 8944 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 22:36:01.946899 8944 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 22:36:01.946931 8944 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.193
I20260925 22:36:01.952155 8944 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:40793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.193.keytab
--principal=kudu/127.6.246.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.246.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.246.254:34691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8944
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.193
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:36:01.953258 8944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:36:01.954308 8944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:36:01.963652 8949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:01.963687 8950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:01.963691 8952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:01.964148 8944 server_base.cc:1034] running on GCE node
I20260925 22:36:01.964564 8944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:36:01.965161 8944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:36:01.966370 8944 hybrid_clock.cc:648] HybridClock initialized: now 1790375761966356 us; error 35 us; skew 500 ppm
Sep 25 22:36:01 dist-test-slave-wjdr krb5kdc[8827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375761, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:36:01.969352 8944 init.cc:377] Logged in from keytab as kudu/127.6.246.193@KRBTEST.COM (short username kudu)
I20260925 22:36:01.970189 8944 webserver.cc:460] Webserver started at http://127.6.246.193:44323/ using document root <none> and password file <none>
I20260925 22:36:01.970711 8944 fs_manager.cc:359] Metadata directory not provided
I20260925 22:36:01.970772 8944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:36:01.970930 8944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:36:01.972931 8944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/minicluster-data/ts-0/data/instance:
uuid: "6a9cd141f38d4c21b0dd6fcbdfbb343f"
format_stamp: "Formatted at 2026-09-25 22:36:01 on dist-test-slave-wjdr"
server_key: "2c89cb8a1b9729e71b2eb8ceda1d9b82"
server_key_iv: "bf21271fc3e3ecfb0e9985f53b0cc751"
server_key_version: "encryptionkey@0"
I20260925 22:36:01.973438 8944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/minicluster-data/ts-0/wal/instance:
uuid: "6a9cd141f38d4c21b0dd6fcbdfbb343f"
format_stamp: "Formatted at 2026-09-25 22:36:01 on dist-test-slave-wjdr"
server_key: "2c89cb8a1b9729e71b2eb8ceda1d9b82"
server_key_iv: "bf21271fc3e3ecfb0e9985f53b0cc751"
server_key_version: "encryptionkey@0"
I20260925 22:36:01.977033 8944 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:36:01.979394 8959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:01.980787 8944 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:36:01.980904 8944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/minicluster-data/ts-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/minicluster-data/ts-0/wal
uuid: "6a9cd141f38d4c21b0dd6fcbdfbb343f"
format_stamp: "Formatted at 2026-09-25 22:36:01 on dist-test-slave-wjdr"
server_key: "2c89cb8a1b9729e71b2eb8ceda1d9b82"
server_key_iv: "bf21271fc3e3ecfb0e9985f53b0cc751"
server_key_version: "encryptionkey@0"
I20260925 22:36:01.981006 8944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:36:02.004987 8944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:36:02.008419 8944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:36:02.008684 8944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:36:02.010216 8944 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:36:02.010268 8944 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:02.010360 8944 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:36:02.010394 8944 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 22:36:02 dist-test-slave-wjdr krb5kdc[8827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375761, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.193@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
I20260925 22:36:02.021365 8944 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.193:36365
I20260925 22:36:02.021397 9074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.193:36365 every 8 connection(s)
I20260925 22:36:02.022534 8944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
I20260925 22:36:02.024725 8921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:02.011061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.193:47513 (local address 127.6.246.254:34691)
0925 22:36:02.011239 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:02.011246 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:02.012098 (+ 852us) server_negotiation.cc:407] Connection header received
0925 22:36:02.013303 (+ 1205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:02.013307 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:02.013365 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:02.013473 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:02.014701 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:02.015244 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:02.016497 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:02.016787 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:02.016909 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:02.020617 (+ 3708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:02.020621 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:02.020623 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:02.020648 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:02.022480 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:02.023137 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:02.023140 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:02.023142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:02.023232 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:02.023677 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:02.023679 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:02.023681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:02.023888 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:02.023970 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:02.024434 (+ 464us) server_negotiation.cc:299] Negotiation successful
0925 22:36:02.024602 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 22:36:02.025492 8968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:02.011464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:34691 (local address 127.6.246.193:47513)
0925 22:36:02.011927 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:02.011977 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:02.013041 (+ 1064us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:02.013649 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:02.013657 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:02.014073 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:02.014551 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:02.014565 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:02.015376 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:02.015380 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:02.016344 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:02.016354 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:02.016951 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:02.016955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:02.017149 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:02.017882 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:02.017907 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:02.020406 (+ 2499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:36:02.022651 (+ 2245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:02.022657 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:02.022672 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:02.023007 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:02.023385 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:02.023389 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:02.023391 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:02.023576 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:02.023980 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:36:02.023984 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:36:02.024207 (+ 223us) client_negotiation.cc:769] Sending connection context
0925 22:36:02.024372 (+ 165us) client_negotiation.cc:240] Negotiation successful
0925 22:36:02.024692 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":137,"threads_started":1}
I20260925 22:36:02.026764 9075 heartbeater.cc:347] Connected to a master server at 127.6.246.254:34691
I20260925 22:36:02.026912 9075 heartbeater.cc:464] Registering TS with master...
I20260925 22:36:02.027467 9075 heartbeater.cc:511] Master 127.6.246.254:34691 requested a full tablet report, sending...
I20260925 22:36:02.028546 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 8944
I20260925 22:36:02.028668 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/minicluster-data/ts-0/wal/instance
I20260925 22:36:02.028997 7131 external_mini_cluster.cc:1428] Setting key 06a3e1a031bd03cd310492e4f037b1a8
I20260925 22:36:02.029289 8871 ts_manager.cc:194] Registered new tserver with Master: 6a9cd141f38d4c21b0dd6fcbdfbb343f (127.6.246.193:36365)
I20260925 22:36:02.030730 8871 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.246.193@KRBTEST.COM'} at 127.6.246.193:47513
I20260925 22:36:02.037271 9081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:02.029153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:34691 (local address 127.6.246.193:41635)
0925 22:36:02.029665 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:02.029679 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:02.029795 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:02.030048 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:02.030051 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:02.030261 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:02.030402 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:02.030409 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:02.031395 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:02.031399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:02.032176 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:02.032188 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:02.032770 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:02.032774 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:02.032875 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:02.033430 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:02.033450 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:02.033776 (+ 326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:36:02.035813 (+ 2037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:02.035815 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:02.035817 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:02.036009 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:02.036308 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:02.036310 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:02.036312 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:02.036397 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:02.036895 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:36:02.036897 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:36:02.036968 (+ 71us) client_negotiation.cc:769] Sending connection context
0925 22:36:02.037045 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 22:36:02.037143 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":96,"threads_started":1}
I20260925 22:36:02.037422 8921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:02.029508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.193:41635 (local address 127.6.246.254:34691)
0925 22:36:02.029634 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:02.029638 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:02.029754 (+ 116us) server_negotiation.cc:407] Connection header received
0925 22:36:02.029890 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:02.029900 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:02.029950 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:02.030033 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:02.030520 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:02.031227 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:02.032307 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:02.032647 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:02.032765 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:02.033923 (+ 1158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:02.033930 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:02.033936 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:02.033966 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:02.035691 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:02.036104 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:02.036110 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:02.036114 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:02.036210 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:02.036517 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:02.036526 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:02.036529 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:02.036788 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:02.036892 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:02.037123 (+ 231us) server_negotiation.cc:299] Negotiation successful
0925 22:36:02.037258 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
WARNING: no policy specified for kudu/127.6.246.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.194.keytab.
Entry for principal kudu/127.6.246.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.194.keytab.
I20260925 22:36:02.061024 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.194:0
--local_ip_for_outbound_sockets=127.6.246.194
--webserver_interface=127.6.246.194
--webserver_port=0
--tserver_master_addrs=127.6.246.254:34691
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.194.keytab
--principal=kudu/127.6.246.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:40793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:36:02.169111 9084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:36:02.169353 9084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:36:02.169397 9084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:36:02.173352 9084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:36:02.173403 9084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:36:02.173440 9084 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 22:36:02.173453 9084 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 22:36:02.173484 9084 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.194
I20260925 22:36:02.178514 9084 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:40793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.194.keytab
--principal=kudu/127.6.246.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.246.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.246.254:34691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9084
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.194
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:36:02.179453 9084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:36:02.180404 9084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:36:02.189699 9092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:02.189693 9089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:02.189693 9090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:02.190287 9084 server_base.cc:1034] running on GCE node
I20260925 22:36:02.190671 9084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:36:02.191337 9084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:36:02.192525 9084 hybrid_clock.cc:648] HybridClock initialized: now 1790375762192486 us; error 44 us; skew 500 ppm
Sep 25 22:36:02 dist-test-slave-wjdr krb5kdc[8827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375762, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:36:02.195204 9084 init.cc:377] Logged in from keytab as kudu/127.6.246.194@KRBTEST.COM (short username kudu)
I20260925 22:36:02.195978 9084 webserver.cc:460] Webserver started at http://127.6.246.194:42801/ using document root <none> and password file <none>
I20260925 22:36:02.196544 9084 fs_manager.cc:359] Metadata directory not provided
I20260925 22:36:02.196606 9084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:36:02.196769 9084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:36:02.198642 9084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/minicluster-data/ts-1/data/instance:
uuid: "5ccf4d68c3db4e05ba0ed000b51d552e"
format_stamp: "Formatted at 2026-09-25 22:36:02 on dist-test-slave-wjdr"
server_key: "c711ce9c202d7c5826bf8c379f9e7b28"
server_key_iv: "d6d07e44189e79ae106cd5f40d359a82"
server_key_version: "encryptionkey@0"
I20260925 22:36:02.199134 9084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/minicluster-data/ts-1/wal/instance:
uuid: "5ccf4d68c3db4e05ba0ed000b51d552e"
format_stamp: "Formatted at 2026-09-25 22:36:02 on dist-test-slave-wjdr"
server_key: "c711ce9c202d7c5826bf8c379f9e7b28"
server_key_iv: "d6d07e44189e79ae106cd5f40d359a82"
server_key_version: "encryptionkey@0"
I20260925 22:36:02.202525 9084 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 22:36:02.204763 9099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:02.205982 9084 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 22:36:02.206111 9084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/minicluster-data/ts-1/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/minicluster-data/ts-1/wal
uuid: "5ccf4d68c3db4e05ba0ed000b51d552e"
format_stamp: "Formatted at 2026-09-25 22:36:02 on dist-test-slave-wjdr"
server_key: "c711ce9c202d7c5826bf8c379f9e7b28"
server_key_iv: "d6d07e44189e79ae106cd5f40d359a82"
server_key_version: "encryptionkey@0"
I20260925 22:36:02.206208 9084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:36:02.219444 9084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:36:02.222241 9084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:36:02.222435 9084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:36:02.223879 9084 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:36:02.223946 9084 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:02.224018 9084 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:36:02.224057 9084 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 22:36:02 dist-test-slave-wjdr krb5kdc[8827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375762, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.194@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
I20260925 22:36:02.235355 9084 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.194:42487
I20260925 22:36:02.235420 9214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.194:42487 every 8 connection(s)
I20260925 22:36:02.236565 9084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
I20260925 22:36:02.236914 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 9084
I20260925 22:36:02.237051 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/minicluster-data/ts-1/wal/instance
I20260925 22:36:02.237361 7131 external_mini_cluster.cc:1428] Setting key ed3be4b60a0756720c95a61db5b45102
I20260925 22:36:02.239331 8921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:02.224757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.194:34371 (local address 127.6.246.254:34691)
0925 22:36:02.224925 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:02.224930 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:02.225896 (+ 966us) server_negotiation.cc:407] Connection header received
0925 22:36:02.227217 (+ 1321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:02.227221 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:02.227293 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:02.227431 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:02.228971 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:02.229765 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:02.231321 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:02.231740 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:02.231868 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:02.235078 (+ 3210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:02.235085 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:02.235092 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:02.235130 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:02.236824 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:02.237479 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:02.237482 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:02.237484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:02.237570 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:02.238132 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:02.238134 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:02.238135 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:02.238335 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:02.238423 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:02.239028 (+ 605us) server_negotiation.cc:299] Negotiation successful
0925 22:36:02.239147 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 22:36:02.240226 9108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:02.225208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:34691 (local address 127.6.246.194:34371)
0925 22:36:02.225665 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:02.225715 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:02.226935 (+ 1220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:02.227654 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:02.227666 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:02.228133 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:02.228814 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:02.228829 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:02.229916 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:02.229919 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:02.231186 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:02.231196 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:02.231946 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:02.231954 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:02.232188 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:02.232863 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:02.232887 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:02.234852 (+ 1965us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:36:02.236972 (+ 2120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:02.236980 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:02.236997 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:02.237362 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:02.237672 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:02.237675 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:02.237677 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:02.237858 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:02.238446 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:36:02.238452 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:36:02.238757 (+ 305us) client_negotiation.cc:769] Sending connection context
0925 22:36:02.238975 (+ 218us) client_negotiation.cc:240] Negotiation successful
0925 22:36:02.239295 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":130,"threads_started":1}
I20260925 22:36:02.241755 9215 heartbeater.cc:347] Connected to a master server at 127.6.246.254:34691
I20260925 22:36:02.241901 9215 heartbeater.cc:464] Registering TS with master...
I20260925 22:36:02.242426 9215 heartbeater.cc:511] Master 127.6.246.254:34691 requested a full tablet report, sending...
I20260925 22:36:02.243408 8872 ts_manager.cc:194] Registered new tserver with Master: 5ccf4d68c3db4e05ba0ed000b51d552e (127.6.246.194:42487)
I20260925 22:36:02.244040 8872 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.246.194@KRBTEST.COM'} at 127.6.246.194:34371
WARNING: no policy specified for kudu/127.6.246.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.195@KRBTEST.COM" created.
I20260925 22:36:02.251950 9222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:02.244034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:34691 (local address 127.6.246.194:46719)
0925 22:36:02.244437 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:02.244453 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:02.244588 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:02.244903 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:02.244908 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:02.245145 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:02.245342 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:02.245353 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:02.246322 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:02.246326 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:02.247307 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:02.247320 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:02.247774 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:02.247778 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:02.247888 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:02.248381 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:02.248401 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:02.248838 (+ 437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:36:02.250529 (+ 1691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:02.250531 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:02.250533 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:02.250708 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:02.250995 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:02.250997 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:02.250998 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:02.251070 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:02.251467 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:36:02.251469 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:36:02.251546 (+ 77us) client_negotiation.cc:769] Sending connection context
0925 22:36:02.251659 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 22:36:02.251787 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"mutex_wait_us":4,"thread_start_us":77,"threads_started":1}
I20260925 22:36:02.251981 8921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:02.244097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.194:46719 (local address 127.6.246.254:34691)
0925 22:36:02.244228 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:02.244234 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:02.244541 (+ 307us) server_negotiation.cc:407] Connection header received
0925 22:36:02.244732 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:02.244736 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:02.244782 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:02.244864 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:02.245473 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:02.246183 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:02.247439 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:02.247650 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:02.247745 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:02.248996 (+ 1251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:02.249000 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:02.249003 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:02.249025 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:02.250438 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:02.250789 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:02.250792 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:02.250793 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:02.250895 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:02.251143 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:02.251145 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:02.251146 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:02.251366 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:02.251483 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:02.251729 (+ 246us) server_negotiation.cc:299] Negotiation successful
0925 22:36:02.251855 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.195.keytab.
Entry for principal kudu/127.6.246.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.195.keytab.
I20260925 22:36:02.267753 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.195:0
--local_ip_for_outbound_sockets=127.6.246.195
--webserver_interface=127.6.246.195
--webserver_port=0
--tserver_master_addrs=127.6.246.254:34691
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.195.keytab
--principal=kudu/127.6.246.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:40793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:36:02.375279 9224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:36:02.375564 9224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:36:02.375641 9224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:36:02.379700 9224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:36:02.379776 9224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:36:02.379843 9224 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 22:36:02.379884 9224 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 22:36:02.379943 9224 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.195
I20260925 22:36:02.385272 9224 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:40793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.195.keytab
--principal=kudu/127.6.246.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.246.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.246.254:34691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9224
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.195
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:36:02.386371 9224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:36:02.387364 9224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:36:02.396636 9230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:02.396643 9229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:02.396643 9232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:02.396697 9224 server_base.cc:1034] running on GCE node
I20260925 22:36:02.397212 9224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:36:02.397748 9224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:36:02.398900 9224 hybrid_clock.cc:648] HybridClock initialized: now 1790375762398889 us; error 24 us; skew 500 ppm
Sep 25 22:36:02 dist-test-slave-wjdr krb5kdc[8827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375762, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:36:02.401546 9224 init.cc:377] Logged in from keytab as kudu/127.6.246.195@KRBTEST.COM (short username kudu)
I20260925 22:36:02.402380 9224 webserver.cc:460] Webserver started at http://127.6.246.195:35305/ using document root <none> and password file <none>
I20260925 22:36:02.402908 9224 fs_manager.cc:359] Metadata directory not provided
I20260925 22:36:02.402990 9224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:36:02.403228 9224 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:36:02.405195 9224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/minicluster-data/ts-2/data/instance:
uuid: "2b55a36d7ff142c183080f141f6892bf"
format_stamp: "Formatted at 2026-09-25 22:36:02 on dist-test-slave-wjdr"
server_key: "c9eb7a6e2e2a9e173c03be4650f0342d"
server_key_iv: "dfba830de4205fd9cce09ffac7f40235"
server_key_version: "encryptionkey@0"
I20260925 22:36:02.405736 9224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/minicluster-data/ts-2/wal/instance:
uuid: "2b55a36d7ff142c183080f141f6892bf"
format_stamp: "Formatted at 2026-09-25 22:36:02 on dist-test-slave-wjdr"
server_key: "c9eb7a6e2e2a9e173c03be4650f0342d"
server_key_iv: "dfba830de4205fd9cce09ffac7f40235"
server_key_version: "encryptionkey@0"
I20260925 22:36:02.409286 9224 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 22:36:02.411549 9239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:02.412717 9224 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 22:36:02.412842 9224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/minicluster-data/ts-2/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/minicluster-data/ts-2/wal
uuid: "2b55a36d7ff142c183080f141f6892bf"
format_stamp: "Formatted at 2026-09-25 22:36:02 on dist-test-slave-wjdr"
server_key: "c9eb7a6e2e2a9e173c03be4650f0342d"
server_key_iv: "dfba830de4205fd9cce09ffac7f40235"
server_key_version: "encryptionkey@0"
I20260925 22:36:02.412942 9224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:36:02.431008 9224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:36:02.434057 9224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:36:02.434257 9224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:36:02.435858 9224 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:36:02.435926 9224 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:02.435990 9224 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:36:02.436026 9224 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 22:36:02 dist-test-slave-wjdr krb5kdc[8827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375762, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.195@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
I20260925 22:36:02.446539 9224 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.195:43427
I20260925 22:36:02.446626 9354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.195:43427 every 8 connection(s)
I20260925 22:36:02.447496 9224 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
I20260925 22:36:02.449756 8921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:02.436502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.195:44385 (local address 127.6.246.254:34691)
0925 22:36:02.436634 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:02.436640 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:02.437986 (+ 1346us) server_negotiation.cc:407] Connection header received
0925 22:36:02.439358 (+ 1372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:02.439361 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:02.439407 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:02.439473 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:02.441142 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:02.441918 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:02.443193 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:02.443537 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:02.443648 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:02.446160 (+ 2512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:02.446164 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:02.446166 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:02.446191 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:02.447668 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:02.448252 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:02.448255 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:02.448257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:02.448344 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:02.448730 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:02.448733 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:02.448735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:02.448922 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:02.449026 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:02.449476 (+ 450us) server_negotiation.cc:299] Negotiation successful
0925 22:36:02.449590 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 22:36:02.450459 9248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:02.437342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:34691 (local address 127.6.246.195:44385)
0925 22:36:02.437814 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:02.437875 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:02.439112 (+ 1237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:02.439693 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:02.439705 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:02.440217 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:02.440985 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:02.441011 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:02.442066 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:02.442069 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:02.443059 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:02.443068 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:02.443681 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:02.443685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:02.443859 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:02.444416 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:02.444443 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:02.446005 (+ 1562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:36:02.447776 (+ 1771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:02.447781 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:02.447795 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:02.448129 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:02.448477 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:02.448480 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:02.448482 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:02.448625 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:02.449023 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:36:02.449029 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:36:02.449240 (+ 211us) client_negotiation.cc:769] Sending connection context
0925 22:36:02.449417 (+ 177us) client_negotiation.cc:240] Negotiation successful
0925 22:36:02.449706 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":101,"threads_started":1}
I20260925 22:36:02.451553 9355 heartbeater.cc:347] Connected to a master server at 127.6.246.254:34691
I20260925 22:36:02.451676 9355 heartbeater.cc:464] Registering TS with master...
I20260925 22:36:02.452178 9355 heartbeater.cc:511] Master 127.6.246.254:34691 requested a full tablet report, sending...
I20260925 22:36:02.453207 8871 ts_manager.cc:194] Registered new tserver with Master: 2b55a36d7ff142c183080f141f6892bf (127.6.246.195:43427)
I20260925 22:36:02.453617 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 9224
I20260925 22:36:02.453768 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/minicluster-data/ts-2/wal/instance
I20260925 22:36:02.453994 8871 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.246.195@KRBTEST.COM'} at 127.6.246.195:44385
I20260925 22:36:02.454140 7131 external_mini_cluster.cc:1428] Setting key e3c150440400b43d1629946c7ada1e07
I20260925 22:36:02.456274 7131 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:36:02.461776 9361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:02.453875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:34691 (local address 127.6.246.195:46247)
0925 22:36:02.454487 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:02.454500 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:02.454627 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:02.455028 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:02.455031 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:02.455246 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:02.455355 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:02.455362 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:02.456845 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:02.456848 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:02.457419 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:02.457426 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:02.457869 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:02.457873 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:02.457960 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:02.458446 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:02.458459 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:02.458746 (+ 287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:36:02.460349 (+ 1603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:02.460354 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:02.460356 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:02.460611 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:02.460889 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:02.460893 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:02.460895 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:02.460979 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:02.461384 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:36:02.461386 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:36:02.461460 (+ 74us) client_negotiation.cc:769] Sending connection context
0925 22:36:02.461548 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 22:36:02.461650 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":77,"threads_started":1}
I20260925 22:36:02.462008 8921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:02.453852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.195:46247 (local address 127.6.246.254:34691)
0925 22:36:02.453961 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:02.453966 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:02.454563 (+ 597us) server_negotiation.cc:407] Connection header received
0925 22:36:02.454801 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:02.454804 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:02.454854 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:02.454928 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:02.456022 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:02.456728 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:02.457534 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:02.457758 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:02.457868 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:02.458887 (+ 1019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:02.458890 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:02.458893 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:02.458915 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:02.460239 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:02.460704 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:02.460709 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:02.460711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:02.460792 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:02.461064 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:02.461069 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:02.461071 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:02.461295 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:02.461400 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:02.461630 (+ 230us) server_negotiation.cc:299] Negotiation successful
0925 22:36:02.461759 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
Sep 25 22:36:02 dist-test-slave-wjdr krb5kdc[8827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 22:36:02 dist-test-slave-wjdr krb5kdc[8827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
I20260925 22:36:02.482239 8921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:02.474300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48472 (local address 127.6.246.254:34691)
0925 22:36:02.474429 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:02.474432 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:02.474443 (+ 11us) server_negotiation.cc:407] Connection header received
0925 22:36:02.474475 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:02.474478 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:02.474511 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:02.474600 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:02.475275 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:02.475778 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:02.476698 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:02.476915 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:02.477022 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:02.479296 (+ 2274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:02.479300 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:02.479302 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:02.479324 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:02.480683 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:02.481072 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:02.481077 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:02.481079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:02.481156 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:02.481435 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:02.481439 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:02.481441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:02.481604 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:02.481743 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:02.481964 (+ 221us) server_negotiation.cc:299] Negotiation successful
0925 22:36:02.482078 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 22:36:02.485128 8871 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48472:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:36:02.488181 8871 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:36:02.502621 9377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:02.497844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44078 (local address 127.6.246.195:43427)
0925 22:36:02.498559 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:02.498571 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:02.498592 (+ 21us) server_negotiation.cc:407] Connection header received
0925 22:36:02.498661 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:02.498665 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:02.498910 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:02.499061 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:36:02.499256 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:02.500072 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:02.501161 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:02.501967 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:02.502086 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:02.502228 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:02.502376 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 22:36:02.502469 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":113,"threads_started":1}
I20260925 22:36:02.502651 9375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:02.496866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.195:43427 (local address 127.0.0.1:44078)
0925 22:36:02.497309 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:02.497322 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:02.497405 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:02.499057 (+ 1652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:02.499060 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:02.499078 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:36:02.499161 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:02.499167 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:02.500195 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:02.500200 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:02.501026 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:02.501035 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:02.502084 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:02.502087 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:02.502187 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:02.502221 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 22:36:02.502346 (+ 125us) client_negotiation.cc:240] Negotiation successful
0925 22:36:02.502507 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"mutex_wait_us":38,"thread_start_us":100,"threads_started":1}
I20260925 22:36:02.502657 9372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:02.496363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.193:36365 (local address 127.0.0.1:39684)
0925 22:36:02.496892 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:02.496942 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:02.497065 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:02.497829 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:02.497837 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:02.497863 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:36:02.497996 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:02.498002 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:02.499620 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:02.499623 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:02.501114 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:02.501121 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:02.502166 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:02.502170 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:02.502275 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:02.502305 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 22:36:02.502356 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 22:36:02.502503 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"mutex_wait_us":86,"thread_start_us":67,"threads_started":1}
I20260925 22:36:02.502835 9376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:02.497457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44572 (local address 127.6.246.194:42487)
0925 22:36:02.497829 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:02.497842 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:02.497864 (+ 22us) server_negotiation.cc:407] Connection header received
0925 22:36:02.497935 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:02.497939 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:02.498143 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:02.498282 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:36:02.498512 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:02.499382 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:02.501246 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:02.502026 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:02.502142 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:02.502279 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:02.502590 (+ 311us) server_negotiation.cc:299] Negotiation successful
0925 22:36:02.502683 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":121,"threads_started":1}
I20260925 22:36:02.503453 9373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:02.496545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39684 (local address 127.6.246.193:36365)
0925 22:36:02.497267 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:02.497285 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:02.497312 (+ 27us) server_negotiation.cc:407] Connection header received
0925 22:36:02.497398 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:02.497403 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:02.497641 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:02.498222 (+ 581us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:36:02.498373 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:02.499512 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:02.501245 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:02.502028 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:02.503002 (+ 974us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:02.503178 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:02.503224 (+ 46us) server_negotiation.cc:299] Negotiation successful
0925 22:36:02.503303 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":85,"threads_started":1}
I20260925 22:36:02.503757 9374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:02.497063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.194:42487 (local address 127.0.0.1:44572)
0925 22:36:02.497570 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:02.497585 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:02.497670 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:02.498277 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:02.498280 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:02.498295 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:36:02.498395 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:02.498400 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:02.499481 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:02.499486 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:02.501103 (+ 1617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:02.501116 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:02.502350 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:02.502355 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:02.502443 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:02.502463 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 22:36:02.503481 (+ 1018us) client_negotiation.cc:240] Negotiation successful
0925 22:36:02.503643 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"mutex_wait_us":112,"thread_start_us":96,"threads_started":1}
I20260925 22:36:02.505867 9009 tablet_service.cc:1467] Processing CreateTablet for tablet e6ae31f78e1543fa902da8d6e9f20701 (DEFAULT_TABLE table=test-table [id=ca9daa50372e4d9c8ff977d9f6c99ac4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:36:02.505838 9289 tablet_service.cc:1467] Processing CreateTablet for tablet e6ae31f78e1543fa902da8d6e9f20701 (DEFAULT_TABLE table=test-table [id=ca9daa50372e4d9c8ff977d9f6c99ac4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:36:02.506742 9009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6ae31f78e1543fa902da8d6e9f20701. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:36:02.506773 9289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6ae31f78e1543fa902da8d6e9f20701. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:36:02.508087 9149 tablet_service.cc:1467] Processing CreateTablet for tablet e6ae31f78e1543fa902da8d6e9f20701 (DEFAULT_TABLE table=test-table [id=ca9daa50372e4d9c8ff977d9f6c99ac4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:36:02.509037 9149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6ae31f78e1543fa902da8d6e9f20701. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:36:02.513988 9378 tablet_bootstrap.cc:492] T e6ae31f78e1543fa902da8d6e9f20701 P 6a9cd141f38d4c21b0dd6fcbdfbb343f: Bootstrap starting.
I20260925 22:36:02.514667 9379 tablet_bootstrap.cc:492] T e6ae31f78e1543fa902da8d6e9f20701 P 2b55a36d7ff142c183080f141f6892bf: Bootstrap starting.
I20260925 22:36:02.516037 9378 tablet_bootstrap.cc:654] T e6ae31f78e1543fa902da8d6e9f20701 P 6a9cd141f38d4c21b0dd6fcbdfbb343f: Neither blocks nor log segments found. Creating new log.
I20260925 22:36:02.516232 9380 tablet_bootstrap.cc:492] T e6ae31f78e1543fa902da8d6e9f20701 P 5ccf4d68c3db4e05ba0ed000b51d552e: Bootstrap starting.
I20260925 22:36:02.516880 9379 tablet_bootstrap.cc:654] T e6ae31f78e1543fa902da8d6e9f20701 P 2b55a36d7ff142c183080f141f6892bf: Neither blocks nor log segments found. Creating new log.
I20260925 22:36:02.517199 9378 log.cc:824] T e6ae31f78e1543fa902da8d6e9f20701 P 6a9cd141f38d4c21b0dd6fcbdfbb343f: Log is configured to *not* fsync() on all Append() calls
I20260925 22:36:02.517604 9379 log.cc:824] T e6ae31f78e1543fa902da8d6e9f20701 P 2b55a36d7ff142c183080f141f6892bf: Log is configured to *not* fsync() on all Append() calls
I20260925 22:36:02.518440 9380 tablet_bootstrap.cc:654] T e6ae31f78e1543fa902da8d6e9f20701 P 5ccf4d68c3db4e05ba0ed000b51d552e: Neither blocks nor log segments found. Creating new log.
I20260925 22:36:02.519233 9378 tablet_bootstrap.cc:492] T e6ae31f78e1543fa902da8d6e9f20701 P 6a9cd141f38d4c21b0dd6fcbdfbb343f: No bootstrap required, opened a new log
I20260925 22:36:02.519217 9380 log.cc:824] T e6ae31f78e1543fa902da8d6e9f20701 P 5ccf4d68c3db4e05ba0ed000b51d552e: Log is configured to *not* fsync() on all Append() calls
I20260925 22:36:02.519435 9378 ts_tablet_manager.cc:1397] T e6ae31f78e1543fa902da8d6e9f20701 P 6a9cd141f38d4c21b0dd6fcbdfbb343f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 22:36:02.519667 9379 tablet_bootstrap.cc:492] T e6ae31f78e1543fa902da8d6e9f20701 P 2b55a36d7ff142c183080f141f6892bf: No bootstrap required, opened a new log
I20260925 22:36:02.519836 9379 ts_tablet_manager.cc:1397] T e6ae31f78e1543fa902da8d6e9f20701 P 2b55a36d7ff142c183080f141f6892bf: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 22:36:02.520896 9380 tablet_bootstrap.cc:492] T e6ae31f78e1543fa902da8d6e9f20701 P 5ccf4d68c3db4e05ba0ed000b51d552e: No bootstrap required, opened a new log
I20260925 22:36:02.521078 9380 ts_tablet_manager.cc:1397] T e6ae31f78e1543fa902da8d6e9f20701 P 5ccf4d68c3db4e05ba0ed000b51d552e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 22:36:02.523342 9378 raft_consensus.cc:348] T e6ae31f78e1543fa902da8d6e9f20701 P 6a9cd141f38d4c21b0dd6fcbdfbb343f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9cd141f38d4c21b0dd6fcbdfbb343f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 36365 } } peers { permanent_uuid: "5ccf4d68c3db4e05ba0ed000b51d552e" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 42487 } } peers { permanent_uuid: "2b55a36d7ff142c183080f141f6892bf" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43427 } }
I20260925 22:36:02.523613 9378 raft_consensus.cc:374] T e6ae31f78e1543fa902da8d6e9f20701 P 6a9cd141f38d4c21b0dd6fcbdfbb343f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:36:02.523461 9379 raft_consensus.cc:348] T e6ae31f78e1543fa902da8d6e9f20701 P 2b55a36d7ff142c183080f141f6892bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9cd141f38d4c21b0dd6fcbdfbb343f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 36365 } } peers { permanent_uuid: "5ccf4d68c3db4e05ba0ed000b51d552e" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 42487 } } peers { permanent_uuid: "2b55a36d7ff142c183080f141f6892bf" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43427 } }
I20260925 22:36:02.523661 9378 raft_consensus.cc:729] T e6ae31f78e1543fa902da8d6e9f20701 P 6a9cd141f38d4c21b0dd6fcbdfbb343f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a9cd141f38d4c21b0dd6fcbdfbb343f, State: Initialized, Role: FOLLOWER
I20260925 22:36:02.523682 9379 raft_consensus.cc:374] T e6ae31f78e1543fa902da8d6e9f20701 P 2b55a36d7ff142c183080f141f6892bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:36:02.523729 9379 raft_consensus.cc:729] T e6ae31f78e1543fa902da8d6e9f20701 P 2b55a36d7ff142c183080f141f6892bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b55a36d7ff142c183080f141f6892bf, State: Initialized, Role: FOLLOWER
I20260925 22:36:02.523667 9380 raft_consensus.cc:348] T e6ae31f78e1543fa902da8d6e9f20701 P 5ccf4d68c3db4e05ba0ed000b51d552e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9cd141f38d4c21b0dd6fcbdfbb343f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 36365 } } peers { permanent_uuid: "5ccf4d68c3db4e05ba0ed000b51d552e" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 42487 } } peers { permanent_uuid: "2b55a36d7ff142c183080f141f6892bf" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43427 } }
I20260925 22:36:02.523924 9380 raft_consensus.cc:374] T e6ae31f78e1543fa902da8d6e9f20701 P 5ccf4d68c3db4e05ba0ed000b51d552e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:36:02.523986 9380 raft_consensus.cc:729] T e6ae31f78e1543fa902da8d6e9f20701 P 5ccf4d68c3db4e05ba0ed000b51d552e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ccf4d68c3db4e05ba0ed000b51d552e, State: Initialized, Role: FOLLOWER
I20260925 22:36:02.524117 9378 consensus_queue.cc:260] T e6ae31f78e1543fa902da8d6e9f20701 P 6a9cd141f38d4c21b0dd6fcbdfbb343f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9cd141f38d4c21b0dd6fcbdfbb343f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 36365 } } peers { permanent_uuid: "5ccf4d68c3db4e05ba0ed000b51d552e" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 42487 } } peers { permanent_uuid: "2b55a36d7ff142c183080f141f6892bf" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43427 } }
I20260925 22:36:02.524117 9379 consensus_queue.cc:260] T e6ae31f78e1543fa902da8d6e9f20701 P 2b55a36d7ff142c183080f141f6892bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9cd141f38d4c21b0dd6fcbdfbb343f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 36365 } } peers { permanent_uuid: "5ccf4d68c3db4e05ba0ed000b51d552e" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 42487 } } peers { permanent_uuid: "2b55a36d7ff142c183080f141f6892bf" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43427 } }
I20260925 22:36:02.524421 9380 consensus_queue.cc:260] T e6ae31f78e1543fa902da8d6e9f20701 P 5ccf4d68c3db4e05ba0ed000b51d552e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9cd141f38d4c21b0dd6fcbdfbb343f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 36365 } } peers { permanent_uuid: "5ccf4d68c3db4e05ba0ed000b51d552e" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 42487 } } peers { permanent_uuid: "2b55a36d7ff142c183080f141f6892bf" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43427 } }
I20260925 22:36:02.525059 9355 heartbeater.cc:503] Master 127.6.246.254:34691 was elected leader, sending a full tablet report...
I20260925 22:36:02.525096 9379 ts_tablet_manager.cc:1428] T e6ae31f78e1543fa902da8d6e9f20701 P 2b55a36d7ff142c183080f141f6892bf: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260925 22:36:02.525096 9378 ts_tablet_manager.cc:1428] T e6ae31f78e1543fa902da8d6e9f20701 P 6a9cd141f38d4c21b0dd6fcbdfbb343f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 22:36:02.525374 9380 ts_tablet_manager.cc:1428] T e6ae31f78e1543fa902da8d6e9f20701 P 5ccf4d68c3db4e05ba0ed000b51d552e: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260925 22:36:02.525671 9215 heartbeater.cc:503] Master 127.6.246.254:34691 was elected leader, sending a full tablet report...
I20260925 22:36:02.526348 9075 heartbeater.cc:503] Master 127.6.246.254:34691 was elected leader, sending a full tablet report...
I20260925 22:36:02.542287 9385 raft_consensus.cc:482] T e6ae31f78e1543fa902da8d6e9f20701 P 6a9cd141f38d4c21b0dd6fcbdfbb343f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:36:02.542480 9385 raft_consensus.cc:504] T e6ae31f78e1543fa902da8d6e9f20701 P 6a9cd141f38d4c21b0dd6fcbdfbb343f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9cd141f38d4c21b0dd6fcbdfbb343f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 36365 } } peers { permanent_uuid: "5ccf4d68c3db4e05ba0ed000b51d552e" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 42487 } } peers { permanent_uuid: "2b55a36d7ff142c183080f141f6892bf" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43427 } }
I20260925 22:36:02.543440 9385 leader_election.cc:288] T e6ae31f78e1543fa902da8d6e9f20701 P 6a9cd141f38d4c21b0dd6fcbdfbb343f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ccf4d68c3db4e05ba0ed000b51d552e (127.6.246.194:42487), 2b55a36d7ff142c183080f141f6892bf (127.6.246.195:43427)
I20260925 22:36:02.547514 9376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:02.544038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.193:40695 (local address 127.6.246.194:42487)
0925 22:36:02.544159 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:02.544164 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:02.544177 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:36:02.544223 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:02.544227 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:02.544273 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:02.544361 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:36:02.544811 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:02.545333 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:02.546348 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:02.546958 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:02.547073 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:02.547125 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:02.547322 (+ 197us) server_negotiation.cc:299] Negotiation successful
0925 22:36:02.547392 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 22:36:02.547545 9388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:02.543665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.194:42487 (local address 127.6.246.193:40695)
0925 22:36:02.543977 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:02.543994 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:02.544092 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:02.544522 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:02.544526 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:02.544570 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:36:02.544689 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:02.544695 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:02.545441 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:02.545444 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:02.546228 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:02.546237 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:02.547079 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:02.547082 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:02.547187 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:02.547206 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 22:36:02.547305 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 22:36:02.547392 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"mutex_wait_us":30,"spinlock_wait_cycles":4736,"thread_start_us":79,"threads_started":1}
I20260925 22:36:02.548121 9169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6ae31f78e1543fa902da8d6e9f20701" candidate_uuid: "6a9cd141f38d4c21b0dd6fcbdfbb343f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ccf4d68c3db4e05ba0ed000b51d552e" is_pre_election: true
I20260925 22:36:02.548341 9387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:02.543665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.195:43427 (local address 127.6.246.193:54545)
0925 22:36:02.544134 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:02.544148 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:02.544250 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:02.544522 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:02.544525 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:02.544568 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:36:02.544694 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:02.544699 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:02.546042 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:02.546046 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:02.547196 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:02.547204 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:02.547947 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:02.547957 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:02.548045 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:02.548066 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 22:36:02.548144 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 22:36:02.548208 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"mutex_wait_us":60,"spinlock_wait_cycles":33280,"thread_start_us":57,"threads_started":1}
I20260925 22:36:02.548377 9377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:02.543804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.193:54545 (local address 127.6.246.195:43427)
0925 22:36:02.543948 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:02.543965 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:02.544218 (+ 253us) server_negotiation.cc:407] Connection header received
0925 22:36:02.544353 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:02.544356 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:02.544403 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:02.544516 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:36:02.545461 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:02.545926 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:02.547309 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:02.547843 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:02.547957 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:02.548009 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:02.548173 (+ 164us) server_negotiation.cc:299] Negotiation successful
0925 22:36:02.548225 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 22:36:02.548470 9169 raft_consensus.cc:2454] T e6ae31f78e1543fa902da8d6e9f20701 P 5ccf4d68c3db4e05ba0ed000b51d552e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a9cd141f38d4c21b0dd6fcbdfbb343f in term 0.
I20260925 22:36:02.548962 8961 leader_election.cc:302] T e6ae31f78e1543fa902da8d6e9f20701 P 6a9cd141f38d4c21b0dd6fcbdfbb343f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ccf4d68c3db4e05ba0ed000b51d552e, 6a9cd141f38d4c21b0dd6fcbdfbb343f; no voters:
I20260925 22:36:02.548977 9308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6ae31f78e1543fa902da8d6e9f20701" candidate_uuid: "6a9cd141f38d4c21b0dd6fcbdfbb343f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b55a36d7ff142c183080f141f6892bf" is_pre_election: true
I20260925 22:36:02.549234 9308 raft_consensus.cc:2454] T e6ae31f78e1543fa902da8d6e9f20701 P 2b55a36d7ff142c183080f141f6892bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a9cd141f38d4c21b0dd6fcbdfbb343f in term 0.
I20260925 22:36:02.549263 9385 raft_consensus.cc:2781] T e6ae31f78e1543fa902da8d6e9f20701 P 6a9cd141f38d4c21b0dd6fcbdfbb343f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:36:02.549327 9385 raft_consensus.cc:482] T e6ae31f78e1543fa902da8d6e9f20701 P 6a9cd141f38d4c21b0dd6fcbdfbb343f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:36:02.549409 9385 raft_consensus.cc:3037] T e6ae31f78e1543fa902da8d6e9f20701 P 6a9cd141f38d4c21b0dd6fcbdfbb343f [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:36:02.550469 9385 raft_consensus.cc:504] T e6ae31f78e1543fa902da8d6e9f20701 P 6a9cd141f38d4c21b0dd6fcbdfbb343f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9cd141f38d4c21b0dd6fcbdfbb343f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 36365 } } peers { permanent_uuid: "5ccf4d68c3db4e05ba0ed000b51d552e" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 42487 } } peers { permanent_uuid: "2b55a36d7ff142c183080f141f6892bf" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43427 } }
I20260925 22:36:02.550807 9385 leader_election.cc:288] T e6ae31f78e1543fa902da8d6e9f20701 P 6a9cd141f38d4c21b0dd6fcbdfbb343f [CANDIDATE]: Term 1 election: Requested vote from peers 5ccf4d68c3db4e05ba0ed000b51d552e (127.6.246.194:42487), 2b55a36d7ff142c183080f141f6892bf (127.6.246.195:43427)
I20260925 22:36:02.551144 9169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6ae31f78e1543fa902da8d6e9f20701" candidate_uuid: "6a9cd141f38d4c21b0dd6fcbdfbb343f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ccf4d68c3db4e05ba0ed000b51d552e"
I20260925 22:36:02.551178 9308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6ae31f78e1543fa902da8d6e9f20701" candidate_uuid: "6a9cd141f38d4c21b0dd6fcbdfbb343f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b55a36d7ff142c183080f141f6892bf"
I20260925 22:36:02.551245 9169 raft_consensus.cc:3037] T e6ae31f78e1543fa902da8d6e9f20701 P 5ccf4d68c3db4e05ba0ed000b51d552e [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:36:02.551385 9308 raft_consensus.cc:3037] T e6ae31f78e1543fa902da8d6e9f20701 P 2b55a36d7ff142c183080f141f6892bf [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:36:02.552428 9308 raft_consensus.cc:2454] T e6ae31f78e1543fa902da8d6e9f20701 P 2b55a36d7ff142c183080f141f6892bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a9cd141f38d4c21b0dd6fcbdfbb343f in term 1.
I20260925 22:36:02.552433 9169 raft_consensus.cc:2454] T e6ae31f78e1543fa902da8d6e9f20701 P 5ccf4d68c3db4e05ba0ed000b51d552e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a9cd141f38d4c21b0dd6fcbdfbb343f in term 1.
I20260925 22:36:02.552749 8961 leader_election.cc:302] T e6ae31f78e1543fa902da8d6e9f20701 P 6a9cd141f38d4c21b0dd6fcbdfbb343f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ccf4d68c3db4e05ba0ed000b51d552e, 6a9cd141f38d4c21b0dd6fcbdfbb343f; no voters:
I20260925 22:36:02.552911 9385 raft_consensus.cc:2781] T e6ae31f78e1543fa902da8d6e9f20701 P 6a9cd141f38d4c21b0dd6fcbdfbb343f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:36:02.553128 9385 raft_consensus.cc:686] T e6ae31f78e1543fa902da8d6e9f20701 P 6a9cd141f38d4c21b0dd6fcbdfbb343f [term 1 LEADER]: Becoming Leader. State: Replica: 6a9cd141f38d4c21b0dd6fcbdfbb343f, State: Running, Role: LEADER
I20260925 22:36:02.553472 9385 consensus_queue.cc:237] T e6ae31f78e1543fa902da8d6e9f20701 P 6a9cd141f38d4c21b0dd6fcbdfbb343f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9cd141f38d4c21b0dd6fcbdfbb343f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 36365 } } peers { permanent_uuid: "5ccf4d68c3db4e05ba0ed000b51d552e" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 42487 } } peers { permanent_uuid: "2b55a36d7ff142c183080f141f6892bf" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43427 } }
I20260925 22:36:02.556699 8870 catalog_manager.cc:5495] T e6ae31f78e1543fa902da8d6e9f20701 P 6a9cd141f38d4c21b0dd6fcbdfbb343f reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a9cd141f38d4c21b0dd6fcbdfbb343f (127.6.246.193). New cstate: current_term: 1 leader_uuid: "6a9cd141f38d4c21b0dd6fcbdfbb343f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9cd141f38d4c21b0dd6fcbdfbb343f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 36365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ccf4d68c3db4e05ba0ed000b51d552e" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 42487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b55a36d7ff142c183080f141f6892bf" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43427 } health_report { overall_health: UNKNOWN } } }
Sep 25 22:36:02 dist-test-slave-wjdr krb5kdc[8827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375761, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.254@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
I20260925 22:36:02.602033 9396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:02.594272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:34691 (local address 127.0.0.1:48474)
0925 22:36:02.594576 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:02.594592 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:02.594709 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:02.594946 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:02.594948 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:02.595348 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:02.595440 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:02.595446 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:02.596163 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:02.596165 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:02.596795 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:02.596804 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:02.597187 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:02.597189 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:02.597247 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:02.597833 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:02.597845 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:02.599190 (+ 1345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:36:02.600712 (+ 1522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:02.600719 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:02.600724 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:02.600938 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:02.601196 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:02.601198 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:02.601199 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:02.601273 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:02.601594 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:36:02.601599 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:36:02.601692 (+ 93us) client_negotiation.cc:769] Sending connection context
0925 22:36:02.601780 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 22:36:02.601884 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":127,"threads_started":1}
I20260925 22:36:02.602051 8921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:02.594300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48474 (local address 127.6.246.254:34691)
0925 22:36:02.594420 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:02.594424 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:02.594671 (+ 247us) server_negotiation.cc:407] Connection header received
0925 22:36:02.594786 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:02.594788 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:02.594830 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:02.594908 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:02.595552 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:02.596070 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:02.596892 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:02.597091 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:02.597168 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:02.599307 (+ 2139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:02.599315 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:02.599317 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:02.599336 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:02.600617 (+ 1281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:02.601040 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:02.601042 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:02.601044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:02.601116 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:02.601349 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:02.601352 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:02.601353 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:02.601510 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:02.601583 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:02.601838 (+ 255us) server_negotiation.cc:299] Negotiation successful
0925 22:36:02.601944 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":42}
I20260925 22:36:02.605785 8870 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.6.246.254@KRBTEST.COM'} at 127.0.0.1:48474:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 22:36:02.606513 8870 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:36:02.611315 9009 tablet_service.cc:1467] Processing CreateTablet for tablet cb6f1f038343440f9654561c40e771bc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=325258d5ad354e5a878133e5a8f4bc82]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 22:36:02.611563 9289 tablet_service.cc:1467] Processing CreateTablet for tablet cb6f1f038343440f9654561c40e771bc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=325258d5ad354e5a878133e5a8f4bc82]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 22:36:02.611557 9149 tablet_service.cc:1467] Processing CreateTablet for tablet cb6f1f038343440f9654561c40e771bc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=325258d5ad354e5a878133e5a8f4bc82]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 22:36:02.612139 9009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb6f1f038343440f9654561c40e771bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:36:02.612139 9289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb6f1f038343440f9654561c40e771bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:36:02.612238 9149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb6f1f038343440f9654561c40e771bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:36:02.614912 9380 tablet_bootstrap.cc:492] T cb6f1f038343440f9654561c40e771bc P 5ccf4d68c3db4e05ba0ed000b51d552e: Bootstrap starting.
I20260925 22:36:02.614993 9378 tablet_bootstrap.cc:492] T cb6f1f038343440f9654561c40e771bc P 6a9cd141f38d4c21b0dd6fcbdfbb343f: Bootstrap starting.
I20260925 22:36:02.614998 9379 tablet_bootstrap.cc:492] T cb6f1f038343440f9654561c40e771bc P 2b55a36d7ff142c183080f141f6892bf: Bootstrap starting.
I20260925 22:36:02.615754 9380 tablet_bootstrap.cc:654] T cb6f1f038343440f9654561c40e771bc P 5ccf4d68c3db4e05ba0ed000b51d552e: Neither blocks nor log segments found. Creating new log.
I20260925 22:36:02.615955 9378 tablet_bootstrap.cc:654] T cb6f1f038343440f9654561c40e771bc P 6a9cd141f38d4c21b0dd6fcbdfbb343f: Neither blocks nor log segments found. Creating new log.
I20260925 22:36:02.615964 9379 tablet_bootstrap.cc:654] T cb6f1f038343440f9654561c40e771bc P 2b55a36d7ff142c183080f141f6892bf: Neither blocks nor log segments found. Creating new log.
I20260925 22:36:02.617375 9379 tablet_bootstrap.cc:492] T cb6f1f038343440f9654561c40e771bc P 2b55a36d7ff142c183080f141f6892bf: No bootstrap required, opened a new log
I20260925 22:36:02.617470 9379 ts_tablet_manager.cc:1397] T cb6f1f038343440f9654561c40e771bc P 2b55a36d7ff142c183080f141f6892bf: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 22:36:02.617687 9380 tablet_bootstrap.cc:492] T cb6f1f038343440f9654561c40e771bc P 5ccf4d68c3db4e05ba0ed000b51d552e: No bootstrap required, opened a new log
I20260925 22:36:02.617784 9380 ts_tablet_manager.cc:1397] T cb6f1f038343440f9654561c40e771bc P 5ccf4d68c3db4e05ba0ed000b51d552e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 22:36:02.617899 9379 raft_consensus.cc:348] T cb6f1f038343440f9654561c40e771bc P 2b55a36d7ff142c183080f141f6892bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9cd141f38d4c21b0dd6fcbdfbb343f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 36365 } } peers { permanent_uuid: "5ccf4d68c3db4e05ba0ed000b51d552e" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 42487 } } peers { permanent_uuid: "2b55a36d7ff142c183080f141f6892bf" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43427 } }
I20260925 22:36:02.618007 9379 raft_consensus.cc:374] T cb6f1f038343440f9654561c40e771bc P 2b55a36d7ff142c183080f141f6892bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:36:02.618031 9379 raft_consensus.cc:729] T cb6f1f038343440f9654561c40e771bc P 2b55a36d7ff142c183080f141f6892bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b55a36d7ff142c183080f141f6892bf, State: Initialized, Role: FOLLOWER
I20260925 22:36:02.618162 9379 consensus_queue.cc:260] T cb6f1f038343440f9654561c40e771bc P 2b55a36d7ff142c183080f141f6892bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9cd141f38d4c21b0dd6fcbdfbb343f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 36365 } } peers { permanent_uuid: "5ccf4d68c3db4e05ba0ed000b51d552e" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 42487 } } peers { permanent_uuid: "2b55a36d7ff142c183080f141f6892bf" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43427 } }
I20260925 22:36:02.618223 9380 raft_consensus.cc:348] T cb6f1f038343440f9654561c40e771bc P 5ccf4d68c3db4e05ba0ed000b51d552e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9cd141f38d4c21b0dd6fcbdfbb343f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 36365 } } peers { permanent_uuid: "5ccf4d68c3db4e05ba0ed000b51d552e" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 42487 } } peers { permanent_uuid: "2b55a36d7ff142c183080f141f6892bf" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43427 } }
I20260925 22:36:02.618345 9380 raft_consensus.cc:374] T cb6f1f038343440f9654561c40e771bc P 5ccf4d68c3db4e05ba0ed000b51d552e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:36:02.618379 9380 raft_consensus.cc:729] T cb6f1f038343440f9654561c40e771bc P 5ccf4d68c3db4e05ba0ed000b51d552e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ccf4d68c3db4e05ba0ed000b51d552e, State: Initialized, Role: FOLLOWER
I20260925 22:36:02.618402 9379 ts_tablet_manager.cc:1428] T cb6f1f038343440f9654561c40e771bc P 2b55a36d7ff142c183080f141f6892bf: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 22:36:02.618419 9384 tablet_replica.cc:439] T cb6f1f038343440f9654561c40e771bc P 2b55a36d7ff142c183080f141f6892bf: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9cd141f38d4c21b0dd6fcbdfbb343f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 36365 } } peers { permanent_uuid: "5ccf4d68c3db4e05ba0ed000b51d552e" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 42487 } } peers { permanent_uuid: "2b55a36d7ff142c183080f141f6892bf" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43427 } } }
I20260925 22:36:02.618491 9380 consensus_queue.cc:260] T cb6f1f038343440f9654561c40e771bc P 5ccf4d68c3db4e05ba0ed000b51d552e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9cd141f38d4c21b0dd6fcbdfbb343f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 36365 } } peers { permanent_uuid: "5ccf4d68c3db4e05ba0ed000b51d552e" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 42487 } } peers { permanent_uuid: "2b55a36d7ff142c183080f141f6892bf" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43427 } }
I20260925 22:36:02.618552 9384 tablet_replica.cc:442] T cb6f1f038343440f9654561c40e771bc P 2b55a36d7ff142c183080f141f6892bf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:36:02.618732 9380 ts_tablet_manager.cc:1428] T cb6f1f038343440f9654561c40e771bc P 5ccf4d68c3db4e05ba0ed000b51d552e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 22:36:02.618741 9386 tablet_replica.cc:439] T cb6f1f038343440f9654561c40e771bc P 5ccf4d68c3db4e05ba0ed000b51d552e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9cd141f38d4c21b0dd6fcbdfbb343f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 36365 } } peers { permanent_uuid: "5ccf4d68c3db4e05ba0ed000b51d552e" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 42487 } } peers { permanent_uuid: "2b55a36d7ff142c183080f141f6892bf" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43427 } } }
I20260925 22:36:02.618927 9386 tablet_replica.cc:442] T cb6f1f038343440f9654561c40e771bc P 5ccf4d68c3db4e05ba0ed000b51d552e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:36:02.619426 9378 tablet_bootstrap.cc:492] T cb6f1f038343440f9654561c40e771bc P 6a9cd141f38d4c21b0dd6fcbdfbb343f: No bootstrap required, opened a new log
I20260925 22:36:02.619527 9378 ts_tablet_manager.cc:1397] T cb6f1f038343440f9654561c40e771bc P 6a9cd141f38d4c21b0dd6fcbdfbb343f: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260925 22:36:02.620052 9378 raft_consensus.cc:348] T cb6f1f038343440f9654561c40e771bc P 6a9cd141f38d4c21b0dd6fcbdfbb343f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9cd141f38d4c21b0dd6fcbdfbb343f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 36365 } } peers { permanent_uuid: "5ccf4d68c3db4e05ba0ed000b51d552e" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 42487 } } peers { permanent_uuid: "2b55a36d7ff142c183080f141f6892bf" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43427 } }
I20260925 22:36:02.620169 9378 raft_consensus.cc:374] T cb6f1f038343440f9654561c40e771bc P 6a9cd141f38d4c21b0dd6fcbdfbb343f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:36:02.620211 9378 raft_consensus.cc:729] T cb6f1f038343440f9654561c40e771bc P 6a9cd141f38d4c21b0dd6fcbdfbb343f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a9cd141f38d4c21b0dd6fcbdfbb343f, State: Initialized, Role: FOLLOWER
I20260925 22:36:02.620327 9378 consensus_queue.cc:260] T cb6f1f038343440f9654561c40e771bc P 6a9cd141f38d4c21b0dd6fcbdfbb343f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9cd141f38d4c21b0dd6fcbdfbb343f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 36365 } } peers { permanent_uuid: "5ccf4d68c3db4e05ba0ed000b51d552e" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 42487 } } peers { permanent_uuid: "2b55a36d7ff142c183080f141f6892bf" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43427 } }
I20260925 22:36:02.620599 9385 tablet_replica.cc:439] T cb6f1f038343440f9654561c40e771bc P 6a9cd141f38d4c21b0dd6fcbdfbb343f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9cd141f38d4c21b0dd6fcbdfbb343f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 36365 } } peers { permanent_uuid: "5ccf4d68c3db4e05ba0ed000b51d552e" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 42487 } } peers { permanent_uuid: "2b55a36d7ff142c183080f141f6892bf" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43427 } } }
I20260925 22:36:02.620730 9385 tablet_replica.cc:442] T cb6f1f038343440f9654561c40e771bc P 6a9cd141f38d4c21b0dd6fcbdfbb343f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:36:02.620930 9378 ts_tablet_manager.cc:1428] T cb6f1f038343440f9654561c40e771bc P 6a9cd141f38d4c21b0dd6fcbdfbb343f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20260925 22:36:02.698557 9356 tablet.cc:2369] T cb6f1f038343440f9654561c40e771bc P 2b55a36d7ff142c183080f141f6892bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 22:36:02.738018 9216 tablet.cc:2369] T cb6f1f038343440f9654561c40e771bc P 5ccf4d68c3db4e05ba0ed000b51d552e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 22:36:02.773931 9076 tablet.cc:2369] T cb6f1f038343440f9654561c40e771bc P 6a9cd141f38d4c21b0dd6fcbdfbb343f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:36:02.809816 9384 raft_consensus.cc:482] T cb6f1f038343440f9654561c40e771bc P 2b55a36d7ff142c183080f141f6892bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:36:02.810022 9384 raft_consensus.cc:504] T cb6f1f038343440f9654561c40e771bc P 2b55a36d7ff142c183080f141f6892bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9cd141f38d4c21b0dd6fcbdfbb343f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 36365 } } peers { permanent_uuid: "5ccf4d68c3db4e05ba0ed000b51d552e" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 42487 } } peers { permanent_uuid: "2b55a36d7ff142c183080f141f6892bf" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43427 } }
I20260925 22:36:02.811124 9384 leader_election.cc:288] T cb6f1f038343440f9654561c40e771bc P 2b55a36d7ff142c183080f141f6892bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a9cd141f38d4c21b0dd6fcbdfbb343f (127.6.246.193:36365), 5ccf4d68c3db4e05ba0ed000b51d552e (127.6.246.194:42487)
I20260925 22:36:02.814884 9248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:02.811377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.193:36365 (local address 127.6.246.195:41043)
0925 22:36:02.811548 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:02.811568 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:02.811679 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:02.811995 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:02.811999 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:02.812016 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:36:02.812136 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:02.812141 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:02.812924 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:02.812930 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:02.813829 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:02.813838 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:02.814456 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:02.814459 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:02.814568 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:02.814595 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 22:36:02.814680 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 22:36:02.814751 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":40}
I20260925 22:36:02.814884 9373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:02.811490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.195:41043 (local address 127.6.246.193:36365)
0925 22:36:02.811634 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:02.811639 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:02.811656 (+ 17us) server_negotiation.cc:407] Connection header received
0925 22:36:02.811813 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:02.811817 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:02.811880 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:02.811964 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:36:02.812247 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:02.812821 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:02.813957 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:02.814340 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:02.814464 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:02.814520 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:02.814695 (+ 175us) server_negotiation.cc:299] Negotiation successful
0925 22:36:02.814762 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 22:36:02.815240 9376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:02.811844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.195:60295 (local address 127.6.246.194:42487)
0925 22:36:02.812015 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:02.812019 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:02.812162 (+ 143us) server_negotiation.cc:407] Connection header received
0925 22:36:02.812286 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:02.812289 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:02.812340 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:02.812418 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:36:02.812703 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:02.813195 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:02.814258 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:02.814645 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:02.814775 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:02.814831 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:02.815049 (+ 218us) server_negotiation.cc:299] Negotiation successful
0925 22:36:02.815125 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260925 22:36:02.815490 9029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb6f1f038343440f9654561c40e771bc" candidate_uuid: "2b55a36d7ff142c183080f141f6892bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a9cd141f38d4c21b0dd6fcbdfbb343f" is_pre_election: true
I20260925 22:36:02.815575 9400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:02.811703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.194:42487 (local address 127.6.246.195:60295)
0925 22:36:02.812075 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:02.812087 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:02.812203 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:02.812440 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:02.812443 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:02.812480 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:36:02.812601 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:02.812607 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:02.813311 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:02.813314 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:02.814140 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:02.814149 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:02.814775 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:02.814779 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:02.814875 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:02.814895 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 22:36:02.815386 (+ 491us) client_negotiation.cc:240] Negotiation successful
0925 22:36:02.815460 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"mutex_wait_us":1,"thread_start_us":127,"threads_started":1}
I20260925 22:36:02.815724 9029 raft_consensus.cc:2454] T cb6f1f038343440f9654561c40e771bc P 6a9cd141f38d4c21b0dd6fcbdfbb343f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b55a36d7ff142c183080f141f6892bf in term 0.
I20260925 22:36:02.816061 9169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb6f1f038343440f9654561c40e771bc" candidate_uuid: "2b55a36d7ff142c183080f141f6892bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ccf4d68c3db4e05ba0ed000b51d552e" is_pre_election: true
I20260925 22:36:02.816197 9169 raft_consensus.cc:2454] T cb6f1f038343440f9654561c40e771bc P 5ccf4d68c3db4e05ba0ed000b51d552e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b55a36d7ff142c183080f141f6892bf in term 0.
I20260925 22:36:02.816226 9241 leader_election.cc:302] T cb6f1f038343440f9654561c40e771bc P 2b55a36d7ff142c183080f141f6892bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b55a36d7ff142c183080f141f6892bf, 6a9cd141f38d4c21b0dd6fcbdfbb343f; no voters:
I20260925 22:36:02.816548 9384 raft_consensus.cc:2781] T cb6f1f038343440f9654561c40e771bc P 2b55a36d7ff142c183080f141f6892bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:36:02.816610 9384 raft_consensus.cc:482] T cb6f1f038343440f9654561c40e771bc P 2b55a36d7ff142c183080f141f6892bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:36:02.816638 9384 raft_consensus.cc:3037] T cb6f1f038343440f9654561c40e771bc P 2b55a36d7ff142c183080f141f6892bf [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:36:02.817724 9384 raft_consensus.cc:504] T cb6f1f038343440f9654561c40e771bc P 2b55a36d7ff142c183080f141f6892bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9cd141f38d4c21b0dd6fcbdfbb343f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 36365 } } peers { permanent_uuid: "5ccf4d68c3db4e05ba0ed000b51d552e" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 42487 } } peers { permanent_uuid: "2b55a36d7ff142c183080f141f6892bf" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43427 } }
I20260925 22:36:02.818075 9384 leader_election.cc:288] T cb6f1f038343440f9654561c40e771bc P 2b55a36d7ff142c183080f141f6892bf [CANDIDATE]: Term 1 election: Requested vote from peers 6a9cd141f38d4c21b0dd6fcbdfbb343f (127.6.246.193:36365), 5ccf4d68c3db4e05ba0ed000b51d552e (127.6.246.194:42487)
I20260925 22:36:02.818460 9029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb6f1f038343440f9654561c40e771bc" candidate_uuid: "2b55a36d7ff142c183080f141f6892bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a9cd141f38d4c21b0dd6fcbdfbb343f"
I20260925 22:36:02.818518 9169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb6f1f038343440f9654561c40e771bc" candidate_uuid: "2b55a36d7ff142c183080f141f6892bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ccf4d68c3db4e05ba0ed000b51d552e"
I20260925 22:36:02.818598 9029 raft_consensus.cc:3037] T cb6f1f038343440f9654561c40e771bc P 6a9cd141f38d4c21b0dd6fcbdfbb343f [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:36:02.818636 9169 raft_consensus.cc:3037] T cb6f1f038343440f9654561c40e771bc P 5ccf4d68c3db4e05ba0ed000b51d552e [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:36:02.819514 9169 raft_consensus.cc:2454] T cb6f1f038343440f9654561c40e771bc P 5ccf4d68c3db4e05ba0ed000b51d552e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b55a36d7ff142c183080f141f6892bf in term 1.
I20260925 22:36:02.819787 9241 leader_election.cc:302] T cb6f1f038343440f9654561c40e771bc P 2b55a36d7ff142c183080f141f6892bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b55a36d7ff142c183080f141f6892bf, 5ccf4d68c3db4e05ba0ed000b51d552e; no voters:
I20260925 22:36:02.819834 9029 raft_consensus.cc:2454] T cb6f1f038343440f9654561c40e771bc P 6a9cd141f38d4c21b0dd6fcbdfbb343f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b55a36d7ff142c183080f141f6892bf in term 1.
I20260925 22:36:02.819976 9384 raft_consensus.cc:2781] T cb6f1f038343440f9654561c40e771bc P 2b55a36d7ff142c183080f141f6892bf [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:36:02.820232 9384 raft_consensus.cc:686] T cb6f1f038343440f9654561c40e771bc P 2b55a36d7ff142c183080f141f6892bf [term 1 LEADER]: Becoming Leader. State: Replica: 2b55a36d7ff142c183080f141f6892bf, State: Running, Role: LEADER
I20260925 22:36:02.820653 9384 consensus_queue.cc:237] T cb6f1f038343440f9654561c40e771bc P 2b55a36d7ff142c183080f141f6892bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9cd141f38d4c21b0dd6fcbdfbb343f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 36365 } } peers { permanent_uuid: "5ccf4d68c3db4e05ba0ed000b51d552e" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 42487 } } peers { permanent_uuid: "2b55a36d7ff142c183080f141f6892bf" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43427 } }
I20260925 22:36:02.822629 9401 tablet_replica.cc:439] T cb6f1f038343440f9654561c40e771bc P 2b55a36d7ff142c183080f141f6892bf: TxnStatusTablet state changed. Reason: New leader 2b55a36d7ff142c183080f141f6892bf. Latest consensus state: current_term: 1 leader_uuid: "2b55a36d7ff142c183080f141f6892bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9cd141f38d4c21b0dd6fcbdfbb343f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 36365 } } peers { permanent_uuid: "5ccf4d68c3db4e05ba0ed000b51d552e" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 42487 } } peers { permanent_uuid: "2b55a36d7ff142c183080f141f6892bf" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43427 } } }
I20260925 22:36:02.822746 9401 tablet_replica.cc:442] T cb6f1f038343440f9654561c40e771bc P 2b55a36d7ff142c183080f141f6892bf: This TxnStatusTablet replica's current role is: LEADER
I20260925 22:36:02.823041 9403 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 22:36:02.823251 9403 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 22:36:02.824038 8871 catalog_manager.cc:5495] T cb6f1f038343440f9654561c40e771bc P 2b55a36d7ff142c183080f141f6892bf reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b55a36d7ff142c183080f141f6892bf (127.6.246.195). New cstate: current_term: 1 leader_uuid: "2b55a36d7ff142c183080f141f6892bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9cd141f38d4c21b0dd6fcbdfbb343f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 36365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ccf4d68c3db4e05ba0ed000b51d552e" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 42487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b55a36d7ff142c183080f141f6892bf" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43427 } health_report { overall_health: HEALTHY } } }
I20260925 22:36:02.853673 9396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:02.849355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.195:43427 (local address 127.0.0.1:44094)
0925 22:36:02.849554 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:02.849568 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:02.849658 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:02.850022 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:02.850024 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:02.850031 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 22:36:02.850122 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:02.850126 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:02.851201 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:02.851204 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:02.851879 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:02.851887 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:02.852522 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:02.852527 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:02.852610 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:02.852653 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 22:36:02.853413 (+ 760us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 22:36:02.853427 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 22:36:02.853497 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 22:36:02.853562 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260925 22:36:02.853722 9377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:02.849467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44094 (local address 127.6.246.195:43427)
0925 22:36:02.849606 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:02.849611 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:02.849627 (+ 16us) server_negotiation.cc:407] Connection header received
0925 22:36:02.849799 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:02.849802 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:02.849867 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:02.850000 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:36:02.850223 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:02.851092 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:02.851985 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:02.852352 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:02.852484 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:02.852784 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:36:02.853293 (+ 509us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:36:02.853406 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:02.853527 (+ 121us) server_negotiation.cc:299] Negotiation successful
0925 22:36:02.853577 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 22:36:02.858680 9029 raft_consensus.cc:1261] T cb6f1f038343440f9654561c40e771bc P 6a9cd141f38d4c21b0dd6fcbdfbb343f [term 1 FOLLOWER]: Refusing update from remote peer 2b55a36d7ff142c183080f141f6892bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:36:02.858682 9169 raft_consensus.cc:1261] T cb6f1f038343440f9654561c40e771bc P 5ccf4d68c3db4e05ba0ed000b51d552e [term 1 FOLLOWER]: Refusing update from remote peer 2b55a36d7ff142c183080f141f6892bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:36:02.859334 9401 consensus_queue.cc:1035] T cb6f1f038343440f9654561c40e771bc P 2b55a36d7ff142c183080f141f6892bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ccf4d68c3db4e05ba0ed000b51d552e" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 42487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:36:02.859517 9384 consensus_queue.cc:1035] T cb6f1f038343440f9654561c40e771bc P 2b55a36d7ff142c183080f141f6892bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a9cd141f38d4c21b0dd6fcbdfbb343f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 36365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:36:02.862673 9385 tablet_replica.cc:439] T cb6f1f038343440f9654561c40e771bc P 6a9cd141f38d4c21b0dd6fcbdfbb343f: TxnStatusTablet state changed. Reason: New leader 2b55a36d7ff142c183080f141f6892bf. Latest consensus state: current_term: 1 leader_uuid: "2b55a36d7ff142c183080f141f6892bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9cd141f38d4c21b0dd6fcbdfbb343f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 36365 } } peers { permanent_uuid: "5ccf4d68c3db4e05ba0ed000b51d552e" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 42487 } } peers { permanent_uuid: "2b55a36d7ff142c183080f141f6892bf" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43427 } } }
I20260925 22:36:02.862794 9385 tablet_replica.cc:442] T cb6f1f038343440f9654561c40e771bc P 6a9cd141f38d4c21b0dd6fcbdfbb343f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:36:02.863880 9386 tablet_replica.cc:439] T cb6f1f038343440f9654561c40e771bc P 5ccf4d68c3db4e05ba0ed000b51d552e: TxnStatusTablet state changed. Reason: New leader 2b55a36d7ff142c183080f141f6892bf. Latest consensus state: current_term: 1 leader_uuid: "2b55a36d7ff142c183080f141f6892bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9cd141f38d4c21b0dd6fcbdfbb343f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 36365 } } peers { permanent_uuid: "5ccf4d68c3db4e05ba0ed000b51d552e" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 42487 } } peers { permanent_uuid: "2b55a36d7ff142c183080f141f6892bf" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43427 } } }
I20260925 22:36:02.863992 9386 tablet_replica.cc:442] T cb6f1f038343440f9654561c40e771bc P 5ccf4d68c3db4e05ba0ed000b51d552e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:36:02.866242 9384 tablet_replica.cc:439] T cb6f1f038343440f9654561c40e771bc P 2b55a36d7ff142c183080f141f6892bf: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2b55a36d7ff142c183080f141f6892bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9cd141f38d4c21b0dd6fcbdfbb343f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 36365 } } peers { permanent_uuid: "5ccf4d68c3db4e05ba0ed000b51d552e" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 42487 } } peers { permanent_uuid: "2b55a36d7ff142c183080f141f6892bf" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43427 } } }
I20260925 22:36:02.866364 9384 tablet_replica.cc:442] T cb6f1f038343440f9654561c40e771bc P 2b55a36d7ff142c183080f141f6892bf: This TxnStatusTablet replica's current role is: LEADER
I20260925 22:36:02.866467 9384 tablet_replica.cc:439] T cb6f1f038343440f9654561c40e771bc P 2b55a36d7ff142c183080f141f6892bf: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2b55a36d7ff142c183080f141f6892bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9cd141f38d4c21b0dd6fcbdfbb343f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 36365 } } peers { permanent_uuid: "5ccf4d68c3db4e05ba0ed000b51d552e" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 42487 } } peers { permanent_uuid: "2b55a36d7ff142c183080f141f6892bf" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43427 } } }
I20260925 22:36:02.866557 9384 tablet_replica.cc:442] T cb6f1f038343440f9654561c40e771bc P 2b55a36d7ff142c183080f141f6892bf: This TxnStatusTablet replica's current role is: LEADER
I20260925 22:36:02.866591 9408 mvcc.cc:205] Tried to move back new op lower bound from 7333379124662566912 to 7333379124516003840. Current Snapshot: MvccSnapshot[applied={T|T < 7333379124662566912}]
I20260925 22:36:02.866930 9406 mvcc.cc:205] Tried to move back new op lower bound from 7333379124662566912 to 7333379124516003840. Current Snapshot: MvccSnapshot[applied={T|T < 7333379124662566912}]
I20260925 22:36:02.867090 9385 tablet_replica.cc:439] T cb6f1f038343440f9654561c40e771bc P 6a9cd141f38d4c21b0dd6fcbdfbb343f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2b55a36d7ff142c183080f141f6892bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9cd141f38d4c21b0dd6fcbdfbb343f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 36365 } } peers { permanent_uuid: "5ccf4d68c3db4e05ba0ed000b51d552e" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 42487 } } peers { permanent_uuid: "2b55a36d7ff142c183080f141f6892bf" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43427 } } }
I20260925 22:36:02.867208 9385 tablet_replica.cc:442] T cb6f1f038343440f9654561c40e771bc P 6a9cd141f38d4c21b0dd6fcbdfbb343f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:36:02.868628 9386 tablet_replica.cc:439] T cb6f1f038343440f9654561c40e771bc P 5ccf4d68c3db4e05ba0ed000b51d552e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2b55a36d7ff142c183080f141f6892bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9cd141f38d4c21b0dd6fcbdfbb343f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 36365 } } peers { permanent_uuid: "5ccf4d68c3db4e05ba0ed000b51d552e" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 42487 } } peers { permanent_uuid: "2b55a36d7ff142c183080f141f6892bf" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43427 } } }
I20260925 22:36:02.868736 9386 tablet_replica.cc:442] T cb6f1f038343440f9654561c40e771bc P 5ccf4d68c3db4e05ba0ed000b51d552e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:36:02.869442 9407 mvcc.cc:205] Tried to move back new op lower bound from 7333379124662566912 to 7333379124516003840. Current Snapshot: MvccSnapshot[applied={T|T < 7333379124662566912}]
I20260925 22:36:02.878561 9373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:02.875048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39696 (local address 127.6.246.193:36365)
0925 22:36:02.875161 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:02.875165 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:02.875177 (+ 12us) server_negotiation.cc:407] Connection header received
0925 22:36:02.875343 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:02.875345 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:02.875390 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:02.875493 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:36:02.875756 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:02.876282 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:02.877195 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:02.877413 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:02.877541 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:02.877804 (+ 263us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:36:02.878174 (+ 370us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:36:02.878291 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:02.878367 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 22:36:02.878430 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 22:36:02.889129 9417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:02.885686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.195:43427 (local address 127.6.246.193:50231)
0925 22:36:02.885973 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:02.885985 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:02.886056 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:02.886292 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:02.886294 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:02.886299 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 22:36:02.886400 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:02.886404 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:02.887333 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:02.887336 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:02.887941 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:02.887949 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:02.888512 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:02.888516 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:02.888583 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:02.888620 (+ 37us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 22:36:02.888901 (+ 281us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 22:36:02.888905 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 22:36:02.888953 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 22:36:02.889010 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":104,"threads_started":1}
I20260925 22:36:02.889130 9377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:02.885833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.193:50231 (local address 127.6.246.195:43427)
0925 22:36:02.886009 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:02.886014 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:02.886028 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:36:02.886147 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:02.886150 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:02.886200 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:02.886279 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:36:02.886492 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:02.887233 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:02.888040 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:02.888369 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:02.888490 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:02.888749 (+ 259us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:36:02.888822 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:36:02.888891 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:02.888963 (+ 72us) server_negotiation.cc:299] Negotiation successful
0925 22:36:02.889006 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260925 22:36:02.895846 9169 raft_consensus.cc:1261] T e6ae31f78e1543fa902da8d6e9f20701 P 5ccf4d68c3db4e05ba0ed000b51d552e [term 1 FOLLOWER]: Refusing update from remote peer 6a9cd141f38d4c21b0dd6fcbdfbb343f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:36:02.895985 9308 raft_consensus.cc:1261] T e6ae31f78e1543fa902da8d6e9f20701 P 2b55a36d7ff142c183080f141f6892bf [term 1 FOLLOWER]: Refusing update from remote peer 6a9cd141f38d4c21b0dd6fcbdfbb343f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:36:02.896327 9389 consensus_queue.cc:1035] T e6ae31f78e1543fa902da8d6e9f20701 P 6a9cd141f38d4c21b0dd6fcbdfbb343f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ccf4d68c3db4e05ba0ed000b51d552e" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 42487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:36:02.896509 9385 consensus_queue.cc:1035] T e6ae31f78e1543fa902da8d6e9f20701 P 6a9cd141f38d4c21b0dd6fcbdfbb343f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b55a36d7ff142c183080f141f6892bf" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:36:02.918454 9373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:02.915109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.195:33349 (local address 127.6.246.193:36365)
0925 22:36:02.915216 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:02.915221 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:02.915327 (+ 106us) server_negotiation.cc:407] Connection header received
0925 22:36:02.915471 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:02.915476 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:02.915524 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:02.915591 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:36:02.915829 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:02.916363 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:02.917344 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:02.917679 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:02.917778 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:02.918054 (+ 276us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:36:02.918134 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:36:02.918202 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:02.918303 (+ 101us) server_negotiation.cc:299] Negotiation successful
0925 22:36:02.918354 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 22:36:02.918491 9361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:02.915063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.193:36365 (local address 127.6.246.195:33349)
0925 22:36:02.915256 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:02.915270 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:02.915361 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:02.915599 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:02.915601 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:02.915608 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 22:36:02.915736 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:02.915742 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:02.916494 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:02.916499 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:02.917231 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:02.917240 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:02.917767 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:02.917770 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:02.917874 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:02.917913 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 22:36:02.918210 (+ 297us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 22:36:02.918217 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 22:36:02.918282 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 22:36:02.918360 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92}
I20260925 22:36:02.941682 8871 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48472:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:36:02.941883 8871 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48472:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:36:02.944597 8871 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5f739135c3c04d60a053abf48962a868: Sending DeleteTablet for 3 replicas of tablet e6ae31f78e1543fa902da8d6e9f20701
I20260925 22:36:02.945523 9009 tablet_service.cc:1514] Processing DeleteTablet for tablet e6ae31f78e1543fa902da8d6e9f20701 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:36:02 UTC) from {username='kudu', principal='kudu/127.6.246.254@KRBTEST.COM'} at 127.0.0.1:39684
I20260925 22:36:02.945544 9149 tablet_service.cc:1514] Processing DeleteTablet for tablet e6ae31f78e1543fa902da8d6e9f20701 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:36:02 UTC) from {username='kudu', principal='kudu/127.6.246.254@KRBTEST.COM'} at 127.0.0.1:44572
I20260925 22:36:02.945990 9289 tablet_service.cc:1514] Processing DeleteTablet for tablet e6ae31f78e1543fa902da8d6e9f20701 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:36:02 UTC) from {username='kudu', principal='kudu/127.6.246.254@KRBTEST.COM'} at 127.0.0.1:44078
I20260925 22:36:02.946458 9426 tablet_replica.cc:330] T e6ae31f78e1543fa902da8d6e9f20701 P 6a9cd141f38d4c21b0dd6fcbdfbb343f: stopping tablet replica
I20260925 22:36:02.946750 9426 raft_consensus.cc:2229] T e6ae31f78e1543fa902da8d6e9f20701 P 6a9cd141f38d4c21b0dd6fcbdfbb343f [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:36:02.946877 8871 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48472:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 22:36:02.947104 9426 raft_consensus.cc:2258] T e6ae31f78e1543fa902da8d6e9f20701 P 6a9cd141f38d4c21b0dd6fcbdfbb343f [term 1 FOLLOWER]: Raft consensus is shut down!
W20260925 22:36:02.947281 8871 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:36:02.947425 9428 tablet_replica.cc:330] T e6ae31f78e1543fa902da8d6e9f20701 P 2b55a36d7ff142c183080f141f6892bf: stopping tablet replica
I20260925 22:36:02.947683 9428 raft_consensus.cc:2229] T e6ae31f78e1543fa902da8d6e9f20701 P 2b55a36d7ff142c183080f141f6892bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:36:02.947840 9428 raft_consensus.cc:2258] T e6ae31f78e1543fa902da8d6e9f20701 P 2b55a36d7ff142c183080f141f6892bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:36:02.949215 9427 tablet_replica.cc:330] T e6ae31f78e1543fa902da8d6e9f20701 P 5ccf4d68c3db4e05ba0ed000b51d552e: stopping tablet replica
I20260925 22:36:02.949524 9427 raft_consensus.cc:2229] T e6ae31f78e1543fa902da8d6e9f20701 P 5ccf4d68c3db4e05ba0ed000b51d552e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:36:02.949777 9427 raft_consensus.cc:2258] T e6ae31f78e1543fa902da8d6e9f20701 P 5ccf4d68c3db4e05ba0ed000b51d552e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:36:02.950809 9426 ts_tablet_manager.cc:1905] T e6ae31f78e1543fa902da8d6e9f20701 P 6a9cd141f38d4c21b0dd6fcbdfbb343f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:36:02.953172 9289 tablet_service.cc:1467] Processing CreateTablet for tablet 325d456c76b54b3897aaba544827ea08 (DEFAULT_TABLE table=test-table [id=d64e351852c04ec8acae024a66ab7291]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:36:02.953189 9009 tablet_service.cc:1467] Processing CreateTablet for tablet 325d456c76b54b3897aaba544827ea08 (DEFAULT_TABLE table=test-table [id=d64e351852c04ec8acae024a66ab7291]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:36:02.953426 9289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 325d456c76b54b3897aaba544827ea08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:36:02.953449 9009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 325d456c76b54b3897aaba544827ea08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:36:02.955739 9379 tablet_bootstrap.cc:492] T 325d456c76b54b3897aaba544827ea08 P 2b55a36d7ff142c183080f141f6892bf: Bootstrap starting.
I20260925 22:36:02.956072 9149 tablet_service.cc:1467] Processing CreateTablet for tablet 325d456c76b54b3897aaba544827ea08 (DEFAULT_TABLE table=test-table [id=d64e351852c04ec8acae024a66ab7291]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:36:02.956327 9149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 325d456c76b54b3897aaba544827ea08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:36:02.956496 9428 ts_tablet_manager.cc:1905] T e6ae31f78e1543fa902da8d6e9f20701 P 2b55a36d7ff142c183080f141f6892bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:36:02.956718 9379 tablet_bootstrap.cc:654] T 325d456c76b54b3897aaba544827ea08 P 2b55a36d7ff142c183080f141f6892bf: Neither blocks nor log segments found. Creating new log.
I20260925 22:36:02.958220 9426 ts_tablet_manager.cc:1918] T e6ae31f78e1543fa902da8d6e9f20701 P 6a9cd141f38d4c21b0dd6fcbdfbb343f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 22:36:02.958302 9426 log.cc:1196] T e6ae31f78e1543fa902da8d6e9f20701 P 6a9cd141f38d4c21b0dd6fcbdfbb343f: Deleting WAL directory at /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/minicluster-data/ts-0/wal/wals/e6ae31f78e1543fa902da8d6e9f20701
I20260925 22:36:02.958587 9426 ts_tablet_manager.cc:1939] T e6ae31f78e1543fa902da8d6e9f20701 P 6a9cd141f38d4c21b0dd6fcbdfbb343f: Deleting consensus metadata
I20260925 22:36:02.958850 9378 tablet_bootstrap.cc:492] T 325d456c76b54b3897aaba544827ea08 P 6a9cd141f38d4c21b0dd6fcbdfbb343f: Bootstrap starting.
I20260925 22:36:02.959085 9428 ts_tablet_manager.cc:1918] T e6ae31f78e1543fa902da8d6e9f20701 P 2b55a36d7ff142c183080f141f6892bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 22:36:02.959162 9428 log.cc:1196] T e6ae31f78e1543fa902da8d6e9f20701 P 2b55a36d7ff142c183080f141f6892bf: Deleting WAL directory at /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/minicluster-data/ts-2/wal/wals/e6ae31f78e1543fa902da8d6e9f20701
I20260925 22:36:02.959581 9428 ts_tablet_manager.cc:1939] T e6ae31f78e1543fa902da8d6e9f20701 P 2b55a36d7ff142c183080f141f6892bf: Deleting consensus metadata
I20260925 22:36:02.959923 9378 tablet_bootstrap.cc:654] T 325d456c76b54b3897aaba544827ea08 P 6a9cd141f38d4c21b0dd6fcbdfbb343f: Neither blocks nor log segments found. Creating new log.
I20260925 22:36:02.960538 9380 tablet_bootstrap.cc:492] T 325d456c76b54b3897aaba544827ea08 P 5ccf4d68c3db4e05ba0ed000b51d552e: Bootstrap starting.
I20260925 22:36:02.961062 9378 tablet_bootstrap.cc:492] T 325d456c76b54b3897aaba544827ea08 P 6a9cd141f38d4c21b0dd6fcbdfbb343f: No bootstrap required, opened a new log
I20260925 22:36:02.961134 9378 ts_tablet_manager.cc:1397] T 325d456c76b54b3897aaba544827ea08 P 6a9cd141f38d4c21b0dd6fcbdfbb343f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 22:36:02.961428 9380 tablet_bootstrap.cc:654] T 325d456c76b54b3897aaba544827ea08 P 5ccf4d68c3db4e05ba0ed000b51d552e: Neither blocks nor log segments found. Creating new log.
I20260925 22:36:02.961552 9378 raft_consensus.cc:348] T 325d456c76b54b3897aaba544827ea08 P 6a9cd141f38d4c21b0dd6fcbdfbb343f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9cd141f38d4c21b0dd6fcbdfbb343f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 36365 } } peers { permanent_uuid: "5ccf4d68c3db4e05ba0ed000b51d552e" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 42487 } } peers { permanent_uuid: "2b55a36d7ff142c183080f141f6892bf" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43427 } }
I20260925 22:36:02.961653 9378 raft_consensus.cc:374] T 325d456c76b54b3897aaba544827ea08 P 6a9cd141f38d4c21b0dd6fcbdfbb343f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:36:02.961684 9378 raft_consensus.cc:729] T 325d456c76b54b3897aaba544827ea08 P 6a9cd141f38d4c21b0dd6fcbdfbb343f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a9cd141f38d4c21b0dd6fcbdfbb343f, State: Initialized, Role: FOLLOWER
I20260925 22:36:02.961807 9378 consensus_queue.cc:260] T 325d456c76b54b3897aaba544827ea08 P 6a9cd141f38d4c21b0dd6fcbdfbb343f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9cd141f38d4c21b0dd6fcbdfbb343f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 36365 } } peers { permanent_uuid: "5ccf4d68c3db4e05ba0ed000b51d552e" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 42487 } } peers { permanent_uuid: "2b55a36d7ff142c183080f141f6892bf" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43427 } }
I20260925 22:36:02.962070 9378 ts_tablet_manager.cc:1428] T 325d456c76b54b3897aaba544827ea08 P 6a9cd141f38d4c21b0dd6fcbdfbb343f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:36:02.963064 8859 catalog_manager.cc:4841] TS 2b55a36d7ff142c183080f141f6892bf (127.6.246.195:43427): tablet e6ae31f78e1543fa902da8d6e9f20701 (table test-table [id=ca9daa50372e4d9c8ff977d9f6c99ac4]) successfully deleted
I20260925 22:36:02.963737 8857 catalog_manager.cc:4841] TS 6a9cd141f38d4c21b0dd6fcbdfbb343f (127.6.246.193:36365): tablet e6ae31f78e1543fa902da8d6e9f20701 (table test-table [id=ca9daa50372e4d9c8ff977d9f6c99ac4]) successfully deleted
I20260925 22:36:02.965619 9427 ts_tablet_manager.cc:1905] T e6ae31f78e1543fa902da8d6e9f20701 P 5ccf4d68c3db4e05ba0ed000b51d552e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:36:02.967515 9379 tablet_bootstrap.cc:492] T 325d456c76b54b3897aaba544827ea08 P 2b55a36d7ff142c183080f141f6892bf: No bootstrap required, opened a new log
I20260925 22:36:02.967607 9379 ts_tablet_manager.cc:1397] T 325d456c76b54b3897aaba544827ea08 P 2b55a36d7ff142c183080f141f6892bf: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20260925 22:36:02.968036 9379 raft_consensus.cc:348] T 325d456c76b54b3897aaba544827ea08 P 2b55a36d7ff142c183080f141f6892bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9cd141f38d4c21b0dd6fcbdfbb343f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 36365 } } peers { permanent_uuid: "5ccf4d68c3db4e05ba0ed000b51d552e" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 42487 } } peers { permanent_uuid: "2b55a36d7ff142c183080f141f6892bf" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43427 } }
I20260925 22:36:02.968147 9379 raft_consensus.cc:374] T 325d456c76b54b3897aaba544827ea08 P 2b55a36d7ff142c183080f141f6892bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:36:02.968183 9379 raft_consensus.cc:729] T 325d456c76b54b3897aaba544827ea08 P 2b55a36d7ff142c183080f141f6892bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b55a36d7ff142c183080f141f6892bf, State: Initialized, Role: FOLLOWER
I20260925 22:36:02.968300 9379 consensus_queue.cc:260] T 325d456c76b54b3897aaba544827ea08 P 2b55a36d7ff142c183080f141f6892bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9cd141f38d4c21b0dd6fcbdfbb343f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 36365 } } peers { permanent_uuid: "5ccf4d68c3db4e05ba0ed000b51d552e" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 42487 } } peers { permanent_uuid: "2b55a36d7ff142c183080f141f6892bf" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43427 } }
I20260925 22:36:02.968585 9379 ts_tablet_manager.cc:1428] T 325d456c76b54b3897aaba544827ea08 P 2b55a36d7ff142c183080f141f6892bf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:36:02.969389 9380 tablet_bootstrap.cc:492] T 325d456c76b54b3897aaba544827ea08 P 5ccf4d68c3db4e05ba0ed000b51d552e: No bootstrap required, opened a new log
I20260925 22:36:02.969480 9380 ts_tablet_manager.cc:1397] T 325d456c76b54b3897aaba544827ea08 P 5ccf4d68c3db4e05ba0ed000b51d552e: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260925 22:36:02.969921 9380 raft_consensus.cc:348] T 325d456c76b54b3897aaba544827ea08 P 5ccf4d68c3db4e05ba0ed000b51d552e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9cd141f38d4c21b0dd6fcbdfbb343f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 36365 } } peers { permanent_uuid: "5ccf4d68c3db4e05ba0ed000b51d552e" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 42487 } } peers { permanent_uuid: "2b55a36d7ff142c183080f141f6892bf" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43427 } }
I20260925 22:36:02.970042 9380 raft_consensus.cc:374] T 325d456c76b54b3897aaba544827ea08 P 5ccf4d68c3db4e05ba0ed000b51d552e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:36:02.970077 9380 raft_consensus.cc:729] T 325d456c76b54b3897aaba544827ea08 P 5ccf4d68c3db4e05ba0ed000b51d552e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ccf4d68c3db4e05ba0ed000b51d552e, State: Initialized, Role: FOLLOWER
I20260925 22:36:02.970198 9380 consensus_queue.cc:260] T 325d456c76b54b3897aaba544827ea08 P 5ccf4d68c3db4e05ba0ed000b51d552e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9cd141f38d4c21b0dd6fcbdfbb343f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 36365 } } peers { permanent_uuid: "5ccf4d68c3db4e05ba0ed000b51d552e" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 42487 } } peers { permanent_uuid: "2b55a36d7ff142c183080f141f6892bf" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43427 } }
I20260925 22:36:02.970451 9380 ts_tablet_manager.cc:1428] T 325d456c76b54b3897aaba544827ea08 P 5ccf4d68c3db4e05ba0ed000b51d552e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:36:02.971351 9427 ts_tablet_manager.cc:1918] T e6ae31f78e1543fa902da8d6e9f20701 P 5ccf4d68c3db4e05ba0ed000b51d552e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 22:36:02.971472 9427 log.cc:1196] T e6ae31f78e1543fa902da8d6e9f20701 P 5ccf4d68c3db4e05ba0ed000b51d552e: Deleting WAL directory at /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/minicluster-data/ts-1/wal/wals/e6ae31f78e1543fa902da8d6e9f20701
I20260925 22:36:02.972043 9427 ts_tablet_manager.cc:1939] T e6ae31f78e1543fa902da8d6e9f20701 P 5ccf4d68c3db4e05ba0ed000b51d552e: Deleting consensus metadata
I20260925 22:36:02.973464 8857 catalog_manager.cc:4841] TS 5ccf4d68c3db4e05ba0ed000b51d552e (127.6.246.194:42487): tablet e6ae31f78e1543fa902da8d6e9f20701 (table test-table [id=ca9daa50372e4d9c8ff977d9f6c99ac4]) successfully deleted
I20260925 22:36:03.156361 9386 raft_consensus.cc:482] T 325d456c76b54b3897aaba544827ea08 P 5ccf4d68c3db4e05ba0ed000b51d552e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:36:03.156512 9386 raft_consensus.cc:504] T 325d456c76b54b3897aaba544827ea08 P 5ccf4d68c3db4e05ba0ed000b51d552e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9cd141f38d4c21b0dd6fcbdfbb343f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 36365 } } peers { permanent_uuid: "5ccf4d68c3db4e05ba0ed000b51d552e" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 42487 } } peers { permanent_uuid: "2b55a36d7ff142c183080f141f6892bf" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43427 } }
I20260925 22:36:03.157685 9386 leader_election.cc:288] T 325d456c76b54b3897aaba544827ea08 P 5ccf4d68c3db4e05ba0ed000b51d552e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a9cd141f38d4c21b0dd6fcbdfbb343f (127.6.246.193:36365), 2b55a36d7ff142c183080f141f6892bf (127.6.246.195:43427)
I20260925 22:36:03.161662 9432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:03.157854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.193:36365 (local address 127.6.246.194:48367)
0925 22:36:03.158159 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:03.158176 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:03.158303 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:03.158584 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:03.158587 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:03.158604 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:36:03.158774 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:03.158781 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:03.159563 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:03.159569 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:03.160588 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:03.160597 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:03.161221 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:03.161225 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:03.161323 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:03.161353 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 22:36:03.161421 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 22:36:03.161476 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":78,"thread_start_us":74,"threads_started":1}
I20260925 22:36:03.161662 9373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:03.158004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.194:48367 (local address 127.6.246.193:36365)
0925 22:36:03.158148 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:03.158152 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:03.158279 (+ 127us) server_negotiation.cc:407] Connection header received
0925 22:36:03.158421 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:03.158424 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:03.158484 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:03.158567 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:36:03.158894 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:03.159445 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:03.160705 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:03.161097 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:03.161218 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:03.161287 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:03.161452 (+ 165us) server_negotiation.cc:299] Negotiation successful
0925 22:36:03.161535 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 22:36:03.161703 9433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:03.157983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.195:43427 (local address 127.6.246.194:39569)
0925 22:36:03.158315 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:03.158331 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:03.158442 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:03.158716 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:03.158719 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:03.158732 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:36:03.158912 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:03.158920 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:03.159628 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:03.159630 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:03.160597 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:03.160607 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:03.161279 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:03.161282 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:03.161389 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:03.161431 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 22:36:03.161508 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 22:36:03.161570 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"mutex_wait_us":115,"thread_start_us":77,"threads_started":1}
I20260925 22:36:03.161725 9377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:03.158082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.194:39569 (local address 127.6.246.195:43427)
0925 22:36:03.158267 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:03.158272 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:03.158432 (+ 160us) server_negotiation.cc:407] Connection header received
0925 22:36:03.158556 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:03.158560 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:03.158611 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:03.158688 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:36:03.159023 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:03.159511 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:03.160734 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:03.161115 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:03.161247 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:03.161302 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:03.161539 (+ 237us) server_negotiation.cc:299] Negotiation successful
0925 22:36:03.161604 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260925 22:36:03.162216 9029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "325d456c76b54b3897aaba544827ea08" candidate_uuid: "5ccf4d68c3db4e05ba0ed000b51d552e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a9cd141f38d4c21b0dd6fcbdfbb343f" is_pre_election: true
I20260925 22:36:03.162251 9308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "325d456c76b54b3897aaba544827ea08" candidate_uuid: "5ccf4d68c3db4e05ba0ed000b51d552e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b55a36d7ff142c183080f141f6892bf" is_pre_election: true
I20260925 22:36:03.162371 9029 raft_consensus.cc:2454] T 325d456c76b54b3897aaba544827ea08 P 6a9cd141f38d4c21b0dd6fcbdfbb343f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ccf4d68c3db4e05ba0ed000b51d552e in term 0.
I20260925 22:36:03.162390 9308 raft_consensus.cc:2454] T 325d456c76b54b3897aaba544827ea08 P 2b55a36d7ff142c183080f141f6892bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ccf4d68c3db4e05ba0ed000b51d552e in term 0.
I20260925 22:36:03.162755 9101 leader_election.cc:302] T 325d456c76b54b3897aaba544827ea08 P 5ccf4d68c3db4e05ba0ed000b51d552e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2b55a36d7ff142c183080f141f6892bf, 5ccf4d68c3db4e05ba0ed000b51d552e, 6a9cd141f38d4c21b0dd6fcbdfbb343f; no voters:
I20260925 22:36:03.163035 9386 raft_consensus.cc:2781] T 325d456c76b54b3897aaba544827ea08 P 5ccf4d68c3db4e05ba0ed000b51d552e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:36:03.163100 9386 raft_consensus.cc:482] T 325d456c76b54b3897aaba544827ea08 P 5ccf4d68c3db4e05ba0ed000b51d552e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:36:03.163128 9386 raft_consensus.cc:3037] T 325d456c76b54b3897aaba544827ea08 P 5ccf4d68c3db4e05ba0ed000b51d552e [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:36:03.164134 9386 raft_consensus.cc:504] T 325d456c76b54b3897aaba544827ea08 P 5ccf4d68c3db4e05ba0ed000b51d552e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9cd141f38d4c21b0dd6fcbdfbb343f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 36365 } } peers { permanent_uuid: "5ccf4d68c3db4e05ba0ed000b51d552e" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 42487 } } peers { permanent_uuid: "2b55a36d7ff142c183080f141f6892bf" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43427 } }
I20260925 22:36:03.164546 9386 leader_election.cc:288] T 325d456c76b54b3897aaba544827ea08 P 5ccf4d68c3db4e05ba0ed000b51d552e [CANDIDATE]: Term 1 election: Requested vote from peers 6a9cd141f38d4c21b0dd6fcbdfbb343f (127.6.246.193:36365), 2b55a36d7ff142c183080f141f6892bf (127.6.246.195:43427)
I20260925 22:36:03.164851 9308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "325d456c76b54b3897aaba544827ea08" candidate_uuid: "5ccf4d68c3db4e05ba0ed000b51d552e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b55a36d7ff142c183080f141f6892bf"
I20260925 22:36:03.164851 9029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "325d456c76b54b3897aaba544827ea08" candidate_uuid: "5ccf4d68c3db4e05ba0ed000b51d552e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a9cd141f38d4c21b0dd6fcbdfbb343f"
I20260925 22:36:03.164973 9308 raft_consensus.cc:3037] T 325d456c76b54b3897aaba544827ea08 P 2b55a36d7ff142c183080f141f6892bf [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:36:03.164973 9029 raft_consensus.cc:3037] T 325d456c76b54b3897aaba544827ea08 P 6a9cd141f38d4c21b0dd6fcbdfbb343f [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:36:03.166038 9308 raft_consensus.cc:2454] T 325d456c76b54b3897aaba544827ea08 P 2b55a36d7ff142c183080f141f6892bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ccf4d68c3db4e05ba0ed000b51d552e in term 1.
I20260925 22:36:03.166038 9029 raft_consensus.cc:2454] T 325d456c76b54b3897aaba544827ea08 P 6a9cd141f38d4c21b0dd6fcbdfbb343f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ccf4d68c3db4e05ba0ed000b51d552e in term 1.
I20260925 22:36:03.166316 9103 leader_election.cc:302] T 325d456c76b54b3897aaba544827ea08 P 5ccf4d68c3db4e05ba0ed000b51d552e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2b55a36d7ff142c183080f141f6892bf, 5ccf4d68c3db4e05ba0ed000b51d552e, 6a9cd141f38d4c21b0dd6fcbdfbb343f; no voters:
I20260925 22:36:03.166510 9386 raft_consensus.cc:2781] T 325d456c76b54b3897aaba544827ea08 P 5ccf4d68c3db4e05ba0ed000b51d552e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:36:03.166689 9386 raft_consensus.cc:686] T 325d456c76b54b3897aaba544827ea08 P 5ccf4d68c3db4e05ba0ed000b51d552e [term 1 LEADER]: Becoming Leader. State: Replica: 5ccf4d68c3db4e05ba0ed000b51d552e, State: Running, Role: LEADER
I20260925 22:36:03.166935 9386 consensus_queue.cc:237] T 325d456c76b54b3897aaba544827ea08 P 5ccf4d68c3db4e05ba0ed000b51d552e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9cd141f38d4c21b0dd6fcbdfbb343f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 36365 } } peers { permanent_uuid: "5ccf4d68c3db4e05ba0ed000b51d552e" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 42487 } } peers { permanent_uuid: "2b55a36d7ff142c183080f141f6892bf" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43427 } }
I20260925 22:36:03.169265 8870 catalog_manager.cc:5495] T 325d456c76b54b3897aaba544827ea08 P 5ccf4d68c3db4e05ba0ed000b51d552e reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ccf4d68c3db4e05ba0ed000b51d552e (127.6.246.194). New cstate: current_term: 1 leader_uuid: "5ccf4d68c3db4e05ba0ed000b51d552e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9cd141f38d4c21b0dd6fcbdfbb343f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 36365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ccf4d68c3db4e05ba0ed000b51d552e" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 42487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b55a36d7ff142c183080f141f6892bf" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43427 } health_report { overall_health: UNKNOWN } } }
I20260925 22:36:03.212106 9376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:03.208531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44574 (local address 127.6.246.194:42487)
0925 22:36:03.208653 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:03.208659 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:03.208675 (+ 16us) server_negotiation.cc:407] Connection header received
0925 22:36:03.208878 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:03.208880 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:03.208936 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:03.209054 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:36:03.209307 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:03.209831 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:03.210650 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:03.210874 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:03.210980 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:03.211240 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:36:03.211694 (+ 454us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:36:03.211819 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:03.211895 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 22:36:03.211966 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 22:36:03.221871 9377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:03.218678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.194:34235 (local address 127.6.246.195:43427)
0925 22:36:03.218786 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:03.218789 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:03.218978 (+ 189us) server_negotiation.cc:407] Connection header received
0925 22:36:03.219026 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:03.219028 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:03.219069 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:03.219140 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:36:03.219662 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:03.220162 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:03.220979 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:03.221190 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:03.221270 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:03.221508 (+ 238us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:36:03.221567 (+ 59us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:36:03.221621 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:03.221711 (+ 90us) server_negotiation.cc:299] Negotiation successful
0925 22:36:03.221751 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 22:36:03.221879 9438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:03.218584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.195:43427 (local address 127.6.246.194:34235)
0925 22:36:03.218872 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:03.218884 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:03.218965 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:03.219158 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:03.219160 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:03.219166 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 22:36:03.219568 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:03.219573 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:03.220252 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:03.220254 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:03.220882 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:03.220890 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:03.221281 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:03.221284 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:03.221362 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:03.221400 (+ 38us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 22:36:03.221635 (+ 235us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 22:36:03.221639 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 22:36:03.221686 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 22:36:03.221741 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":105,"threads_started":1}
I20260925 22:36:03.227099 9308 raft_consensus.cc:1261] T 325d456c76b54b3897aaba544827ea08 P 2b55a36d7ff142c183080f141f6892bf [term 1 FOLLOWER]: Refusing update from remote peer 5ccf4d68c3db4e05ba0ed000b51d552e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:36:03.227244 9029 raft_consensus.cc:1261] T 325d456c76b54b3897aaba544827ea08 P 6a9cd141f38d4c21b0dd6fcbdfbb343f [term 1 FOLLOWER]: Refusing update from remote peer 5ccf4d68c3db4e05ba0ed000b51d552e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:36:03.227584 9434 consensus_queue.cc:1035] T 325d456c76b54b3897aaba544827ea08 P 5ccf4d68c3db4e05ba0ed000b51d552e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b55a36d7ff142c183080f141f6892bf" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:36:03.227823 9386 consensus_queue.cc:1035] T 325d456c76b54b3897aaba544827ea08 P 5ccf4d68c3db4e05ba0ed000b51d552e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a9cd141f38d4c21b0dd6fcbdfbb343f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 36365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:36:03.245798 9361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:03.242095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.194:42487 (local address 127.6.246.195:41501)
0925 22:36:03.242211 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:03.242227 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:03.242321 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:03.242667 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:03.242670 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:03.242680 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 22:36:03.242781 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:03.242787 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:03.243776 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:03.243780 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:03.244650 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:03.244660 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:03.245103 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:03.245106 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:03.245183 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:03.245213 (+ 30us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 22:36:03.245521 (+ 308us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 22:36:03.245528 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 22:36:03.245597 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 22:36:03.245676 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260925 22:36:03.246110 9376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:03.242132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.195:41501 (local address 127.6.246.194:42487)
0925 22:36:03.242476 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:03.242480 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:03.242492 (+ 12us) server_negotiation.cc:407] Connection header received
0925 22:36:03.242531 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:03.242533 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:03.242577 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:03.242657 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:36:03.242931 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:03.243657 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:03.244781 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:03.244991 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:03.245075 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:03.245359 (+ 284us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:36:03.245433 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:36:03.245490 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:03.245954 (+ 464us) server_negotiation.cc:299] Negotiation successful
0925 22:36:03.246006 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268}
W20260925 22:36:03.246721 9407 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 22:36:04.818064 8872 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48472:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:36:04.818269 8872 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48472:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:36:04.821300 8872 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5f739135c3c04d60a053abf48962a868: Sending DeleteTablet for 3 replicas of tablet 325d456c76b54b3897aaba544827ea08
I20260925 22:36:04.822077 9009 tablet_service.cc:1514] Processing DeleteTablet for tablet 325d456c76b54b3897aaba544827ea08 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:36:04 UTC) from {username='kudu', principal='kudu/127.6.246.254@KRBTEST.COM'} at 127.0.0.1:39684
I20260925 22:36:04.822096 9149 tablet_service.cc:1514] Processing DeleteTablet for tablet 325d456c76b54b3897aaba544827ea08 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:36:04 UTC) from {username='kudu', principal='kudu/127.6.246.254@KRBTEST.COM'} at 127.0.0.1:44572
I20260925 22:36:04.822098 9289 tablet_service.cc:1514] Processing DeleteTablet for tablet 325d456c76b54b3897aaba544827ea08 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:36:04 UTC) from {username='kudu', principal='kudu/127.6.246.254@KRBTEST.COM'} at 127.0.0.1:44078
I20260925 22:36:04.822526 9469 tablet_replica.cc:330] T 325d456c76b54b3897aaba544827ea08 P 5ccf4d68c3db4e05ba0ed000b51d552e: stopping tablet replica
I20260925 22:36:04.822695 9469 raft_consensus.cc:2229] T 325d456c76b54b3897aaba544827ea08 P 5ccf4d68c3db4e05ba0ed000b51d552e [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:36:04.822707 9470 tablet_replica.cc:330] T 325d456c76b54b3897aaba544827ea08 P 6a9cd141f38d4c21b0dd6fcbdfbb343f: stopping tablet replica
I20260925 22:36:04.822835 9470 raft_consensus.cc:2229] T 325d456c76b54b3897aaba544827ea08 P 6a9cd141f38d4c21b0dd6fcbdfbb343f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:36:04.822962 9470 raft_consensus.cc:2258] T 325d456c76b54b3897aaba544827ea08 P 6a9cd141f38d4c21b0dd6fcbdfbb343f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:36:04.823123 9469 raft_consensus.cc:2258] T 325d456c76b54b3897aaba544827ea08 P 5ccf4d68c3db4e05ba0ed000b51d552e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:36:04.823402 9468 tablet_replica.cc:330] T 325d456c76b54b3897aaba544827ea08 P 2b55a36d7ff142c183080f141f6892bf: stopping tablet replica
I20260925 22:36:04.823541 9468 raft_consensus.cc:2229] T 325d456c76b54b3897aaba544827ea08 P 2b55a36d7ff142c183080f141f6892bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:36:04.823668 9468 raft_consensus.cc:2258] T 325d456c76b54b3897aaba544827ea08 P 2b55a36d7ff142c183080f141f6892bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:36:04.823737 9470 ts_tablet_manager.cc:1905] T 325d456c76b54b3897aaba544827ea08 P 6a9cd141f38d4c21b0dd6fcbdfbb343f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:36:04.824393 9468 ts_tablet_manager.cc:1905] T 325d456c76b54b3897aaba544827ea08 P 2b55a36d7ff142c183080f141f6892bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:36:04.826664 9469 ts_tablet_manager.cc:1905] T 325d456c76b54b3897aaba544827ea08 P 5ccf4d68c3db4e05ba0ed000b51d552e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:36:04.827492 9470 ts_tablet_manager.cc:1918] T 325d456c76b54b3897aaba544827ea08 P 6a9cd141f38d4c21b0dd6fcbdfbb343f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 22:36:04.827600 9470 log.cc:1196] T 325d456c76b54b3897aaba544827ea08 P 6a9cd141f38d4c21b0dd6fcbdfbb343f: Deleting WAL directory at /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/minicluster-data/ts-0/wal/wals/325d456c76b54b3897aaba544827ea08
I20260925 22:36:04.827683 9468 ts_tablet_manager.cc:1918] T 325d456c76b54b3897aaba544827ea08 P 2b55a36d7ff142c183080f141f6892bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 22:36:04.827768 9468 log.cc:1196] T 325d456c76b54b3897aaba544827ea08 P 2b55a36d7ff142c183080f141f6892bf: Deleting WAL directory at /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/minicluster-data/ts-2/wal/wals/325d456c76b54b3897aaba544827ea08
I20260925 22:36:04.827910 9470 ts_tablet_manager.cc:1939] T 325d456c76b54b3897aaba544827ea08 P 6a9cd141f38d4c21b0dd6fcbdfbb343f: Deleting consensus metadata
I20260925 22:36:04.828090 9468 ts_tablet_manager.cc:1939] T 325d456c76b54b3897aaba544827ea08 P 2b55a36d7ff142c183080f141f6892bf: Deleting consensus metadata
I20260925 22:36:04.828754 8857 catalog_manager.cc:4841] TS 6a9cd141f38d4c21b0dd6fcbdfbb343f (127.6.246.193:36365): tablet 325d456c76b54b3897aaba544827ea08 (table test-table [id=d64e351852c04ec8acae024a66ab7291]) successfully deleted
I20260925 22:36:04.828896 8859 catalog_manager.cc:4841] TS 2b55a36d7ff142c183080f141f6892bf (127.6.246.195:43427): tablet 325d456c76b54b3897aaba544827ea08 (table test-table [id=d64e351852c04ec8acae024a66ab7291]) successfully deleted
I20260925 22:36:04.829932 9469 ts_tablet_manager.cc:1918] T 325d456c76b54b3897aaba544827ea08 P 5ccf4d68c3db4e05ba0ed000b51d552e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 22:36:04.830022 9469 log.cc:1196] T 325d456c76b54b3897aaba544827ea08 P 5ccf4d68c3db4e05ba0ed000b51d552e: Deleting WAL directory at /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/minicluster-data/ts-1/wal/wals/325d456c76b54b3897aaba544827ea08
I20260925 22:36:04.830325 9469 ts_tablet_manager.cc:1939] T 325d456c76b54b3897aaba544827ea08 P 5ccf4d68c3db4e05ba0ed000b51d552e: Deleting consensus metadata
I20260925 22:36:04.831085 8857 catalog_manager.cc:4841] TS 5ccf4d68c3db4e05ba0ed000b51d552e (127.6.246.194:42487): tablet 325d456c76b54b3897aaba544827ea08 (table test-table [id=d64e351852c04ec8acae024a66ab7291]) successfully deleted
Sep 25 22:36:04 dist-test-slave-wjdr krb5kdc[8827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375764, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 22:36:04 dist-test-slave-wjdr krb5kdc[8827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375764, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
I20260925 22:36:04.860929 9478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:04.845803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48486 (local address 127.6.246.254:34691)
0925 22:36:04.846160 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:04.846174 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:04.846208 (+ 34us) server_negotiation.cc:407] Connection header received
0925 22:36:04.846418 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:04.846427 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:04.846497 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:04.846619 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:04.847437 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:04.848085 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:04.849098 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:04.849360 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:04.849494 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:04.857509 (+ 8015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:04.857513 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:04.857515 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:04.857544 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:04.859206 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:04.859649 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:04.859653 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:04.859654 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:04.859758 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:04.860018 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:04.860021 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:04.860023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:04.860244 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:04.860364 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:04.860636 (+ 272us) server_negotiation.cc:299] Negotiation successful
0925 22:36:04.860753 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":112,"threads_started":1}
I20260925 22:36:04.864125 8872 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48486:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:36:04.864750 8872 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:36:04.870471 9009 tablet_service.cc:1467] Processing CreateTablet for tablet 548f3a3323074a8d8ab80fb25536046a (DEFAULT_TABLE table=test-table [id=6a4c01fc065b4ae0bcebe961fa031b6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:36:04.870471 9149 tablet_service.cc:1467] Processing CreateTablet for tablet 548f3a3323074a8d8ab80fb25536046a (DEFAULT_TABLE table=test-table [id=6a4c01fc065b4ae0bcebe961fa031b6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:36:04.870862 9149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 548f3a3323074a8d8ab80fb25536046a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:36:04.870862 9009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 548f3a3323074a8d8ab80fb25536046a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:36:04.872838 9289 tablet_service.cc:1467] Processing CreateTablet for tablet 548f3a3323074a8d8ab80fb25536046a (DEFAULT_TABLE table=test-table [id=6a4c01fc065b4ae0bcebe961fa031b6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:36:04.873217 9289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 548f3a3323074a8d8ab80fb25536046a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:36:04.874022 9480 tablet_bootstrap.cc:492] T 548f3a3323074a8d8ab80fb25536046a P 5ccf4d68c3db4e05ba0ed000b51d552e: Bootstrap starting.
I20260925 22:36:04.874269 9479 tablet_bootstrap.cc:492] T 548f3a3323074a8d8ab80fb25536046a P 6a9cd141f38d4c21b0dd6fcbdfbb343f: Bootstrap starting.
I20260925 22:36:04.875183 9480 tablet_bootstrap.cc:654] T 548f3a3323074a8d8ab80fb25536046a P 5ccf4d68c3db4e05ba0ed000b51d552e: Neither blocks nor log segments found. Creating new log.
I20260925 22:36:04.875475 9479 tablet_bootstrap.cc:654] T 548f3a3323074a8d8ab80fb25536046a P 6a9cd141f38d4c21b0dd6fcbdfbb343f: Neither blocks nor log segments found. Creating new log.
I20260925 22:36:04.876653 9480 tablet_bootstrap.cc:492] T 548f3a3323074a8d8ab80fb25536046a P 5ccf4d68c3db4e05ba0ed000b51d552e: No bootstrap required, opened a new log
I20260925 22:36:04.876754 9480 ts_tablet_manager.cc:1397] T 548f3a3323074a8d8ab80fb25536046a P 5ccf4d68c3db4e05ba0ed000b51d552e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 22:36:04.876832 9479 tablet_bootstrap.cc:492] T 548f3a3323074a8d8ab80fb25536046a P 6a9cd141f38d4c21b0dd6fcbdfbb343f: No bootstrap required, opened a new log
I20260925 22:36:04.876868 9482 tablet_bootstrap.cc:492] T 548f3a3323074a8d8ab80fb25536046a P 2b55a36d7ff142c183080f141f6892bf: Bootstrap starting.
I20260925 22:36:04.876920 9479 ts_tablet_manager.cc:1397] T 548f3a3323074a8d8ab80fb25536046a P 6a9cd141f38d4c21b0dd6fcbdfbb343f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 22:36:04.877211 9480 raft_consensus.cc:348] T 548f3a3323074a8d8ab80fb25536046a P 5ccf4d68c3db4e05ba0ed000b51d552e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9cd141f38d4c21b0dd6fcbdfbb343f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 36365 } } peers { permanent_uuid: "5ccf4d68c3db4e05ba0ed000b51d552e" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 42487 } } peers { permanent_uuid: "2b55a36d7ff142c183080f141f6892bf" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43427 } }
I20260925 22:36:04.877351 9480 raft_consensus.cc:374] T 548f3a3323074a8d8ab80fb25536046a P 5ccf4d68c3db4e05ba0ed000b51d552e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:36:04.877388 9480 raft_consensus.cc:729] T 548f3a3323074a8d8ab80fb25536046a P 5ccf4d68c3db4e05ba0ed000b51d552e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ccf4d68c3db4e05ba0ed000b51d552e, State: Initialized, Role: FOLLOWER
I20260925 22:36:04.877344 9479 raft_consensus.cc:348] T 548f3a3323074a8d8ab80fb25536046a P 6a9cd141f38d4c21b0dd6fcbdfbb343f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9cd141f38d4c21b0dd6fcbdfbb343f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 36365 } } peers { permanent_uuid: "5ccf4d68c3db4e05ba0ed000b51d552e" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 42487 } } peers { permanent_uuid: "2b55a36d7ff142c183080f141f6892bf" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43427 } }
I20260925 22:36:04.877455 9479 raft_consensus.cc:374] T 548f3a3323074a8d8ab80fb25536046a P 6a9cd141f38d4c21b0dd6fcbdfbb343f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:36:04.877488 9479 raft_consensus.cc:729] T 548f3a3323074a8d8ab80fb25536046a P 6a9cd141f38d4c21b0dd6fcbdfbb343f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a9cd141f38d4c21b0dd6fcbdfbb343f, State: Initialized, Role: FOLLOWER
I20260925 22:36:04.877549 9480 consensus_queue.cc:260] T 548f3a3323074a8d8ab80fb25536046a P 5ccf4d68c3db4e05ba0ed000b51d552e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9cd141f38d4c21b0dd6fcbdfbb343f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 36365 } } peers { permanent_uuid: "5ccf4d68c3db4e05ba0ed000b51d552e" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 42487 } } peers { permanent_uuid: "2b55a36d7ff142c183080f141f6892bf" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43427 } }
I20260925 22:36:04.877600 9479 consensus_queue.cc:260] T 548f3a3323074a8d8ab80fb25536046a P 6a9cd141f38d4c21b0dd6fcbdfbb343f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9cd141f38d4c21b0dd6fcbdfbb343f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 36365 } } peers { permanent_uuid: "5ccf4d68c3db4e05ba0ed000b51d552e" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 42487 } } peers { permanent_uuid: "2b55a36d7ff142c183080f141f6892bf" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43427 } }
I20260925 22:36:04.877856 9480 ts_tablet_manager.cc:1428] T 548f3a3323074a8d8ab80fb25536046a P 5ccf4d68c3db4e05ba0ed000b51d552e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:36:04.877945 9479 ts_tablet_manager.cc:1428] T 548f3a3323074a8d8ab80fb25536046a P 6a9cd141f38d4c21b0dd6fcbdfbb343f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:36:04.877991 9482 tablet_bootstrap.cc:654] T 548f3a3323074a8d8ab80fb25536046a P 2b55a36d7ff142c183080f141f6892bf: Neither blocks nor log segments found. Creating new log.
I20260925 22:36:04.879241 9482 tablet_bootstrap.cc:492] T 548f3a3323074a8d8ab80fb25536046a P 2b55a36d7ff142c183080f141f6892bf: No bootstrap required, opened a new log
I20260925 22:36:04.879360 9482 ts_tablet_manager.cc:1397] T 548f3a3323074a8d8ab80fb25536046a P 2b55a36d7ff142c183080f141f6892bf: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 22:36:04.879850 9482 raft_consensus.cc:348] T 548f3a3323074a8d8ab80fb25536046a P 2b55a36d7ff142c183080f141f6892bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9cd141f38d4c21b0dd6fcbdfbb343f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 36365 } } peers { permanent_uuid: "5ccf4d68c3db4e05ba0ed000b51d552e" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 42487 } } peers { permanent_uuid: "2b55a36d7ff142c183080f141f6892bf" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43427 } }
I20260925 22:36:04.880005 9482 raft_consensus.cc:374] T 548f3a3323074a8d8ab80fb25536046a P 2b55a36d7ff142c183080f141f6892bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:36:04.880039 9482 raft_consensus.cc:729] T 548f3a3323074a8d8ab80fb25536046a P 2b55a36d7ff142c183080f141f6892bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b55a36d7ff142c183080f141f6892bf, State: Initialized, Role: FOLLOWER
I20260925 22:36:04.880167 9482 consensus_queue.cc:260] T 548f3a3323074a8d8ab80fb25536046a P 2b55a36d7ff142c183080f141f6892bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9cd141f38d4c21b0dd6fcbdfbb343f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 36365 } } peers { permanent_uuid: "5ccf4d68c3db4e05ba0ed000b51d552e" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 42487 } } peers { permanent_uuid: "2b55a36d7ff142c183080f141f6892bf" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43427 } }
I20260925 22:36:04.880434 9482 ts_tablet_manager.cc:1428] T 548f3a3323074a8d8ab80fb25536046a P 2b55a36d7ff142c183080f141f6892bf: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 22:36:04.992903 9484 raft_consensus.cc:482] T 548f3a3323074a8d8ab80fb25536046a P 6a9cd141f38d4c21b0dd6fcbdfbb343f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:36:04.993116 9484 raft_consensus.cc:504] T 548f3a3323074a8d8ab80fb25536046a P 6a9cd141f38d4c21b0dd6fcbdfbb343f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9cd141f38d4c21b0dd6fcbdfbb343f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 36365 } } peers { permanent_uuid: "5ccf4d68c3db4e05ba0ed000b51d552e" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 42487 } } peers { permanent_uuid: "2b55a36d7ff142c183080f141f6892bf" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43427 } }
I20260925 22:36:04.993686 9484 leader_election.cc:288] T 548f3a3323074a8d8ab80fb25536046a P 6a9cd141f38d4c21b0dd6fcbdfbb343f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ccf4d68c3db4e05ba0ed000b51d552e (127.6.246.194:42487), 2b55a36d7ff142c183080f141f6892bf (127.6.246.195:43427)
I20260925 22:36:04.994230 9169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "548f3a3323074a8d8ab80fb25536046a" candidate_uuid: "6a9cd141f38d4c21b0dd6fcbdfbb343f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ccf4d68c3db4e05ba0ed000b51d552e" is_pre_election: true
I20260925 22:36:04.994297 9308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "548f3a3323074a8d8ab80fb25536046a" candidate_uuid: "6a9cd141f38d4c21b0dd6fcbdfbb343f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b55a36d7ff142c183080f141f6892bf" is_pre_election: true
I20260925 22:36:04.994407 9169 raft_consensus.cc:2454] T 548f3a3323074a8d8ab80fb25536046a P 5ccf4d68c3db4e05ba0ed000b51d552e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a9cd141f38d4c21b0dd6fcbdfbb343f in term 0.
I20260925 22:36:04.994519 9308 raft_consensus.cc:2454] T 548f3a3323074a8d8ab80fb25536046a P 2b55a36d7ff142c183080f141f6892bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a9cd141f38d4c21b0dd6fcbdfbb343f in term 0.
I20260925 22:36:04.994782 8961 leader_election.cc:302] T 548f3a3323074a8d8ab80fb25536046a P 6a9cd141f38d4c21b0dd6fcbdfbb343f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ccf4d68c3db4e05ba0ed000b51d552e, 6a9cd141f38d4c21b0dd6fcbdfbb343f; no voters:
I20260925 22:36:04.994992 9484 raft_consensus.cc:2781] T 548f3a3323074a8d8ab80fb25536046a P 6a9cd141f38d4c21b0dd6fcbdfbb343f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:36:04.995083 9484 raft_consensus.cc:482] T 548f3a3323074a8d8ab80fb25536046a P 6a9cd141f38d4c21b0dd6fcbdfbb343f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:36:04.995112 9484 raft_consensus.cc:3037] T 548f3a3323074a8d8ab80fb25536046a P 6a9cd141f38d4c21b0dd6fcbdfbb343f [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:36:04.996455 9484 raft_consensus.cc:504] T 548f3a3323074a8d8ab80fb25536046a P 6a9cd141f38d4c21b0dd6fcbdfbb343f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9cd141f38d4c21b0dd6fcbdfbb343f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 36365 } } peers { permanent_uuid: "5ccf4d68c3db4e05ba0ed000b51d552e" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 42487 } } peers { permanent_uuid: "2b55a36d7ff142c183080f141f6892bf" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43427 } }
I20260925 22:36:04.996912 9484 leader_election.cc:288] T 548f3a3323074a8d8ab80fb25536046a P 6a9cd141f38d4c21b0dd6fcbdfbb343f [CANDIDATE]: Term 1 election: Requested vote from peers 5ccf4d68c3db4e05ba0ed000b51d552e (127.6.246.194:42487), 2b55a36d7ff142c183080f141f6892bf (127.6.246.195:43427)
I20260925 22:36:04.997329 9308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "548f3a3323074a8d8ab80fb25536046a" candidate_uuid: "6a9cd141f38d4c21b0dd6fcbdfbb343f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b55a36d7ff142c183080f141f6892bf"
I20260925 22:36:04.997458 9308 raft_consensus.cc:3037] T 548f3a3323074a8d8ab80fb25536046a P 2b55a36d7ff142c183080f141f6892bf [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:36:04.997395 9169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "548f3a3323074a8d8ab80fb25536046a" candidate_uuid: "6a9cd141f38d4c21b0dd6fcbdfbb343f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ccf4d68c3db4e05ba0ed000b51d552e"
I20260925 22:36:04.997577 9169 raft_consensus.cc:3037] T 548f3a3323074a8d8ab80fb25536046a P 5ccf4d68c3db4e05ba0ed000b51d552e [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:36:04.998628 9308 raft_consensus.cc:2454] T 548f3a3323074a8d8ab80fb25536046a P 2b55a36d7ff142c183080f141f6892bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a9cd141f38d4c21b0dd6fcbdfbb343f in term 1.
I20260925 22:36:04.998788 9169 raft_consensus.cc:2454] T 548f3a3323074a8d8ab80fb25536046a P 5ccf4d68c3db4e05ba0ed000b51d552e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a9cd141f38d4c21b0dd6fcbdfbb343f in term 1.
I20260925 22:36:04.998963 8963 leader_election.cc:302] T 548f3a3323074a8d8ab80fb25536046a P 6a9cd141f38d4c21b0dd6fcbdfbb343f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b55a36d7ff142c183080f141f6892bf, 6a9cd141f38d4c21b0dd6fcbdfbb343f; no voters:
I20260925 22:36:04.999150 9484 raft_consensus.cc:2781] T 548f3a3323074a8d8ab80fb25536046a P 6a9cd141f38d4c21b0dd6fcbdfbb343f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:36:04.999406 9484 raft_consensus.cc:686] T 548f3a3323074a8d8ab80fb25536046a P 6a9cd141f38d4c21b0dd6fcbdfbb343f [term 1 LEADER]: Becoming Leader. State: Replica: 6a9cd141f38d4c21b0dd6fcbdfbb343f, State: Running, Role: LEADER
I20260925 22:36:04.999603 9484 consensus_queue.cc:237] T 548f3a3323074a8d8ab80fb25536046a P 6a9cd141f38d4c21b0dd6fcbdfbb343f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9cd141f38d4c21b0dd6fcbdfbb343f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 36365 } } peers { permanent_uuid: "5ccf4d68c3db4e05ba0ed000b51d552e" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 42487 } } peers { permanent_uuid: "2b55a36d7ff142c183080f141f6892bf" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43427 } }
I20260925 22:36:05.001639 8870 catalog_manager.cc:5495] T 548f3a3323074a8d8ab80fb25536046a P 6a9cd141f38d4c21b0dd6fcbdfbb343f reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a9cd141f38d4c21b0dd6fcbdfbb343f (127.6.246.193). New cstate: current_term: 1 leader_uuid: "6a9cd141f38d4c21b0dd6fcbdfbb343f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9cd141f38d4c21b0dd6fcbdfbb343f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 36365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ccf4d68c3db4e05ba0ed000b51d552e" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 42487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b55a36d7ff142c183080f141f6892bf" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43427 } health_report { overall_health: UNKNOWN } } }
I20260925 22:36:05.039265 9489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:05.035062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39710 (local address 127.6.246.193:36365)
0925 22:36:05.035369 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:05.035377 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:05.035391 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:36:05.035438 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:05.035440 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:05.035498 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:05.035582 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:36:05.035903 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:05.036604 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:05.037989 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:05.038378 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:05.038473 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:05.038745 (+ 272us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:36:05.038846 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:36:05.038916 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:05.039012 (+ 96us) server_negotiation.cc:299] Negotiation successful
0925 22:36:05.039075 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":152,"threads_started":1}
I20260925 22:36:05.046662 9308 raft_consensus.cc:1261] T 548f3a3323074a8d8ab80fb25536046a P 2b55a36d7ff142c183080f141f6892bf [term 1 FOLLOWER]: Refusing update from remote peer 6a9cd141f38d4c21b0dd6fcbdfbb343f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:36:05.046662 9169 raft_consensus.cc:1261] T 548f3a3323074a8d8ab80fb25536046a P 5ccf4d68c3db4e05ba0ed000b51d552e [term 1 FOLLOWER]: Refusing update from remote peer 6a9cd141f38d4c21b0dd6fcbdfbb343f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:36:05.047196 9484 consensus_queue.cc:1035] T 548f3a3323074a8d8ab80fb25536046a P 6a9cd141f38d4c21b0dd6fcbdfbb343f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ccf4d68c3db4e05ba0ed000b51d552e" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 42487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:36:05.047387 9486 consensus_queue.cc:1035] T 548f3a3323074a8d8ab80fb25536046a P 6a9cd141f38d4c21b0dd6fcbdfbb343f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b55a36d7ff142c183080f141f6892bf" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:36:05.080772 8870 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48486:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:36:05.080986 8870 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48486:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:36:05.083230 8870 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5f739135c3c04d60a053abf48962a868: Sending DeleteTablet for 3 replicas of tablet 548f3a3323074a8d8ab80fb25536046a
I20260925 22:36:05.083810 9009 tablet_service.cc:1514] Processing DeleteTablet for tablet 548f3a3323074a8d8ab80fb25536046a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:36:05 UTC) from {username='kudu', principal='kudu/127.6.246.254@KRBTEST.COM'} at 127.0.0.1:39684
I20260925 22:36:05.083927 9289 tablet_service.cc:1514] Processing DeleteTablet for tablet 548f3a3323074a8d8ab80fb25536046a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:36:05 UTC) from {username='kudu', principal='kudu/127.6.246.254@KRBTEST.COM'} at 127.0.0.1:44078
I20260925 22:36:05.084028 9149 tablet_service.cc:1514] Processing DeleteTablet for tablet 548f3a3323074a8d8ab80fb25536046a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:36:05 UTC) from {username='kudu', principal='kudu/127.6.246.254@KRBTEST.COM'} at 127.0.0.1:44572
I20260925 22:36:05.084105 9470 tablet_replica.cc:330] T 548f3a3323074a8d8ab80fb25536046a P 6a9cd141f38d4c21b0dd6fcbdfbb343f: stopping tablet replica
I20260925 22:36:05.084177 9468 tablet_replica.cc:330] T 548f3a3323074a8d8ab80fb25536046a P 2b55a36d7ff142c183080f141f6892bf: stopping tablet replica
I20260925 22:36:05.084221 9469 tablet_replica.cc:330] T 548f3a3323074a8d8ab80fb25536046a P 5ccf4d68c3db4e05ba0ed000b51d552e: stopping tablet replica
I20260925 22:36:05.084254 9470 raft_consensus.cc:2229] T 548f3a3323074a8d8ab80fb25536046a P 6a9cd141f38d4c21b0dd6fcbdfbb343f [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:36:05.084295 9468 raft_consensus.cc:2229] T 548f3a3323074a8d8ab80fb25536046a P 2b55a36d7ff142c183080f141f6892bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:36:05.084332 9469 raft_consensus.cc:2229] T 548f3a3323074a8d8ab80fb25536046a P 5ccf4d68c3db4e05ba0ed000b51d552e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:36:05.084398 9468 raft_consensus.cc:2258] T 548f3a3323074a8d8ab80fb25536046a P 2b55a36d7ff142c183080f141f6892bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:36:05.084479 9469 raft_consensus.cc:2258] T 548f3a3323074a8d8ab80fb25536046a P 5ccf4d68c3db4e05ba0ed000b51d552e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:36:05.084630 9470 raft_consensus.cc:2258] T 548f3a3323074a8d8ab80fb25536046a P 6a9cd141f38d4c21b0dd6fcbdfbb343f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:36:05.085186 8870 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48486:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 22:36:05.085390 9469 ts_tablet_manager.cc:1905] T 548f3a3323074a8d8ab80fb25536046a P 5ccf4d68c3db4e05ba0ed000b51d552e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:36:05.085402 9468 ts_tablet_manager.cc:1905] T 548f3a3323074a8d8ab80fb25536046a P 2b55a36d7ff142c183080f141f6892bf: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 22:36:05.085737 8870 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:36:05.088064 9468 ts_tablet_manager.cc:1918] T 548f3a3323074a8d8ab80fb25536046a P 2b55a36d7ff142c183080f141f6892bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 22:36:05.088181 9468 log.cc:1196] T 548f3a3323074a8d8ab80fb25536046a P 2b55a36d7ff142c183080f141f6892bf: Deleting WAL directory at /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/minicluster-data/ts-2/wal/wals/548f3a3323074a8d8ab80fb25536046a
I20260925 22:36:05.088438 9469 ts_tablet_manager.cc:1918] T 548f3a3323074a8d8ab80fb25536046a P 5ccf4d68c3db4e05ba0ed000b51d552e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 22:36:05.088543 9468 ts_tablet_manager.cc:1939] T 548f3a3323074a8d8ab80fb25536046a P 2b55a36d7ff142c183080f141f6892bf: Deleting consensus metadata
I20260925 22:36:05.088573 9469 log.cc:1196] T 548f3a3323074a8d8ab80fb25536046a P 5ccf4d68c3db4e05ba0ed000b51d552e: Deleting WAL directory at /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/minicluster-data/ts-1/wal/wals/548f3a3323074a8d8ab80fb25536046a
I20260925 22:36:05.088925 9469 ts_tablet_manager.cc:1939] T 548f3a3323074a8d8ab80fb25536046a P 5ccf4d68c3db4e05ba0ed000b51d552e: Deleting consensus metadata
I20260925 22:36:05.089668 9470 ts_tablet_manager.cc:1905] T 548f3a3323074a8d8ab80fb25536046a P 6a9cd141f38d4c21b0dd6fcbdfbb343f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:36:05.089815 8857 catalog_manager.cc:4841] TS 5ccf4d68c3db4e05ba0ed000b51d552e (127.6.246.194:42487): tablet 548f3a3323074a8d8ab80fb25536046a (table test-table [id=6a4c01fc065b4ae0bcebe961fa031b6b]) successfully deleted
I20260925 22:36:05.092685 9470 ts_tablet_manager.cc:1918] T 548f3a3323074a8d8ab80fb25536046a P 6a9cd141f38d4c21b0dd6fcbdfbb343f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 22:36:05.092692 8859 catalog_manager.cc:4841] TS 2b55a36d7ff142c183080f141f6892bf (127.6.246.195:43427): tablet 548f3a3323074a8d8ab80fb25536046a (table test-table [id=6a4c01fc065b4ae0bcebe961fa031b6b]) successfully deleted
I20260925 22:36:05.092790 9470 log.cc:1196] T 548f3a3323074a8d8ab80fb25536046a P 6a9cd141f38d4c21b0dd6fcbdfbb343f: Deleting WAL directory at /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/minicluster-data/ts-0/wal/wals/548f3a3323074a8d8ab80fb25536046a
I20260925 22:36:05.093076 9470 ts_tablet_manager.cc:1939] T 548f3a3323074a8d8ab80fb25536046a P 6a9cd141f38d4c21b0dd6fcbdfbb343f: Deleting consensus metadata
I20260925 22:36:05.093930 8857 catalog_manager.cc:4841] TS 6a9cd141f38d4c21b0dd6fcbdfbb343f (127.6.246.193:36365): tablet 548f3a3323074a8d8ab80fb25536046a (table test-table [id=6a4c01fc065b4ae0bcebe961fa031b6b]) successfully deleted
I20260925 22:36:05.094069 9009 tablet_service.cc:1467] Processing CreateTablet for tablet 2a67abb34f31418db9f6b9f3018ab368 (DEFAULT_TABLE table=test-table [id=7929c1f8fa074e4b8755a6d083404f1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:36:05.094131 9149 tablet_service.cc:1467] Processing CreateTablet for tablet 2a67abb34f31418db9f6b9f3018ab368 (DEFAULT_TABLE table=test-table [id=7929c1f8fa074e4b8755a6d083404f1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:36:05.094372 9009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a67abb34f31418db9f6b9f3018ab368. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:36:05.094403 9149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a67abb34f31418db9f6b9f3018ab368. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:36:05.096298 9479 tablet_bootstrap.cc:492] T 2a67abb34f31418db9f6b9f3018ab368 P 6a9cd141f38d4c21b0dd6fcbdfbb343f: Bootstrap starting.
I20260925 22:36:05.096412 9480 tablet_bootstrap.cc:492] T 2a67abb34f31418db9f6b9f3018ab368 P 5ccf4d68c3db4e05ba0ed000b51d552e: Bootstrap starting.
I20260925 22:36:05.097425 9479 tablet_bootstrap.cc:654] T 2a67abb34f31418db9f6b9f3018ab368 P 6a9cd141f38d4c21b0dd6fcbdfbb343f: Neither blocks nor log segments found. Creating new log.
I20260925 22:36:05.097425 9480 tablet_bootstrap.cc:654] T 2a67abb34f31418db9f6b9f3018ab368 P 5ccf4d68c3db4e05ba0ed000b51d552e: Neither blocks nor log segments found. Creating new log.
I20260925 22:36:05.098317 9289 tablet_service.cc:1467] Processing CreateTablet for tablet 2a67abb34f31418db9f6b9f3018ab368 (DEFAULT_TABLE table=test-table [id=7929c1f8fa074e4b8755a6d083404f1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:36:05.098659 9289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a67abb34f31418db9f6b9f3018ab368. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:36:05.098737 9480 tablet_bootstrap.cc:492] T 2a67abb34f31418db9f6b9f3018ab368 P 5ccf4d68c3db4e05ba0ed000b51d552e: No bootstrap required, opened a new log
I20260925 22:36:05.098748 9479 tablet_bootstrap.cc:492] T 2a67abb34f31418db9f6b9f3018ab368 P 6a9cd141f38d4c21b0dd6fcbdfbb343f: No bootstrap required, opened a new log
I20260925 22:36:05.098816 9480 ts_tablet_manager.cc:1397] T 2a67abb34f31418db9f6b9f3018ab368 P 5ccf4d68c3db4e05ba0ed000b51d552e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 22:36:05.098830 9479 ts_tablet_manager.cc:1397] T 2a67abb34f31418db9f6b9f3018ab368 P 6a9cd141f38d4c21b0dd6fcbdfbb343f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 22:36:05.099294 9480 raft_consensus.cc:348] T 2a67abb34f31418db9f6b9f3018ab368 P 5ccf4d68c3db4e05ba0ed000b51d552e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9cd141f38d4c21b0dd6fcbdfbb343f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 36365 } } peers { permanent_uuid: "5ccf4d68c3db4e05ba0ed000b51d552e" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 42487 } } peers { permanent_uuid: "2b55a36d7ff142c183080f141f6892bf" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43427 } }
I20260925 22:36:05.099308 9479 raft_consensus.cc:348] T 2a67abb34f31418db9f6b9f3018ab368 P 6a9cd141f38d4c21b0dd6fcbdfbb343f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9cd141f38d4c21b0dd6fcbdfbb343f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 36365 } } peers { permanent_uuid: "5ccf4d68c3db4e05ba0ed000b51d552e" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 42487 } } peers { permanent_uuid: "2b55a36d7ff142c183080f141f6892bf" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43427 } }
I20260925 22:36:05.099423 9480 raft_consensus.cc:374] T 2a67abb34f31418db9f6b9f3018ab368 P 5ccf4d68c3db4e05ba0ed000b51d552e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:36:05.099424 9479 raft_consensus.cc:374] T 2a67abb34f31418db9f6b9f3018ab368 P 6a9cd141f38d4c21b0dd6fcbdfbb343f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:36:05.099464 9480 raft_consensus.cc:729] T 2a67abb34f31418db9f6b9f3018ab368 P 5ccf4d68c3db4e05ba0ed000b51d552e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ccf4d68c3db4e05ba0ed000b51d552e, State: Initialized, Role: FOLLOWER
I20260925 22:36:05.099464 9479 raft_consensus.cc:729] T 2a67abb34f31418db9f6b9f3018ab368 P 6a9cd141f38d4c21b0dd6fcbdfbb343f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a9cd141f38d4c21b0dd6fcbdfbb343f, State: Initialized, Role: FOLLOWER
I20260925 22:36:05.099608 9480 consensus_queue.cc:260] T 2a67abb34f31418db9f6b9f3018ab368 P 5ccf4d68c3db4e05ba0ed000b51d552e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9cd141f38d4c21b0dd6fcbdfbb343f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 36365 } } peers { permanent_uuid: "5ccf4d68c3db4e05ba0ed000b51d552e" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 42487 } } peers { permanent_uuid: "2b55a36d7ff142c183080f141f6892bf" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43427 } }
I20260925 22:36:05.099608 9479 consensus_queue.cc:260] T 2a67abb34f31418db9f6b9f3018ab368 P 6a9cd141f38d4c21b0dd6fcbdfbb343f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9cd141f38d4c21b0dd6fcbdfbb343f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 36365 } } peers { permanent_uuid: "5ccf4d68c3db4e05ba0ed000b51d552e" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 42487 } } peers { permanent_uuid: "2b55a36d7ff142c183080f141f6892bf" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43427 } }
I20260925 22:36:05.099918 9479 ts_tablet_manager.cc:1428] T 2a67abb34f31418db9f6b9f3018ab368 P 6a9cd141f38d4c21b0dd6fcbdfbb343f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 22:36:05.099938 9480 ts_tablet_manager.cc:1428] T 2a67abb34f31418db9f6b9f3018ab368 P 5ccf4d68c3db4e05ba0ed000b51d552e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:36:05.102656 9482 tablet_bootstrap.cc:492] T 2a67abb34f31418db9f6b9f3018ab368 P 2b55a36d7ff142c183080f141f6892bf: Bootstrap starting.
I20260925 22:36:05.103577 9482 tablet_bootstrap.cc:654] T 2a67abb34f31418db9f6b9f3018ab368 P 2b55a36d7ff142c183080f141f6892bf: Neither blocks nor log segments found. Creating new log.
I20260925 22:36:05.105050 9482 tablet_bootstrap.cc:492] T 2a67abb34f31418db9f6b9f3018ab368 P 2b55a36d7ff142c183080f141f6892bf: No bootstrap required, opened a new log
I20260925 22:36:05.105157 9482 ts_tablet_manager.cc:1397] T 2a67abb34f31418db9f6b9f3018ab368 P 2b55a36d7ff142c183080f141f6892bf: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 22:36:05.105583 9482 raft_consensus.cc:348] T 2a67abb34f31418db9f6b9f3018ab368 P 2b55a36d7ff142c183080f141f6892bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9cd141f38d4c21b0dd6fcbdfbb343f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 36365 } } peers { permanent_uuid: "5ccf4d68c3db4e05ba0ed000b51d552e" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 42487 } } peers { permanent_uuid: "2b55a36d7ff142c183080f141f6892bf" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43427 } }
I20260925 22:36:05.105682 9482 raft_consensus.cc:374] T 2a67abb34f31418db9f6b9f3018ab368 P 2b55a36d7ff142c183080f141f6892bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:36:05.105703 9482 raft_consensus.cc:729] T 2a67abb34f31418db9f6b9f3018ab368 P 2b55a36d7ff142c183080f141f6892bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b55a36d7ff142c183080f141f6892bf, State: Initialized, Role: FOLLOWER
I20260925 22:36:05.105806 9482 consensus_queue.cc:260] T 2a67abb34f31418db9f6b9f3018ab368 P 2b55a36d7ff142c183080f141f6892bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9cd141f38d4c21b0dd6fcbdfbb343f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 36365 } } peers { permanent_uuid: "5ccf4d68c3db4e05ba0ed000b51d552e" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 42487 } } peers { permanent_uuid: "2b55a36d7ff142c183080f141f6892bf" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43427 } }
I20260925 22:36:05.106076 9482 ts_tablet_manager.cc:1428] T 2a67abb34f31418db9f6b9f3018ab368 P 2b55a36d7ff142c183080f141f6892bf: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 22:36:05.319841 9449 raft_consensus.cc:482] T 2a67abb34f31418db9f6b9f3018ab368 P 2b55a36d7ff142c183080f141f6892bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:36:05.320030 9449 raft_consensus.cc:504] T 2a67abb34f31418db9f6b9f3018ab368 P 2b55a36d7ff142c183080f141f6892bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9cd141f38d4c21b0dd6fcbdfbb343f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 36365 } } peers { permanent_uuid: "5ccf4d68c3db4e05ba0ed000b51d552e" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 42487 } } peers { permanent_uuid: "2b55a36d7ff142c183080f141f6892bf" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43427 } }
I20260925 22:36:05.320722 9449 leader_election.cc:288] T 2a67abb34f31418db9f6b9f3018ab368 P 2b55a36d7ff142c183080f141f6892bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a9cd141f38d4c21b0dd6fcbdfbb343f (127.6.246.193:36365), 5ccf4d68c3db4e05ba0ed000b51d552e (127.6.246.194:42487)
I20260925 22:36:05.321388 9029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a67abb34f31418db9f6b9f3018ab368" candidate_uuid: "2b55a36d7ff142c183080f141f6892bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a9cd141f38d4c21b0dd6fcbdfbb343f" is_pre_election: true
I20260925 22:36:05.321427 9169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a67abb34f31418db9f6b9f3018ab368" candidate_uuid: "2b55a36d7ff142c183080f141f6892bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ccf4d68c3db4e05ba0ed000b51d552e" is_pre_election: true
I20260925 22:36:05.321604 9029 raft_consensus.cc:2454] T 2a67abb34f31418db9f6b9f3018ab368 P 6a9cd141f38d4c21b0dd6fcbdfbb343f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b55a36d7ff142c183080f141f6892bf in term 0.
I20260925 22:36:05.321619 9169 raft_consensus.cc:2454] T 2a67abb34f31418db9f6b9f3018ab368 P 5ccf4d68c3db4e05ba0ed000b51d552e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b55a36d7ff142c183080f141f6892bf in term 0.
I20260925 22:36:05.321978 9241 leader_election.cc:302] T 2a67abb34f31418db9f6b9f3018ab368 P 2b55a36d7ff142c183080f141f6892bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b55a36d7ff142c183080f141f6892bf, 5ccf4d68c3db4e05ba0ed000b51d552e; no voters:
I20260925 22:36:05.322204 9449 raft_consensus.cc:2781] T 2a67abb34f31418db9f6b9f3018ab368 P 2b55a36d7ff142c183080f141f6892bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:36:05.322263 9449 raft_consensus.cc:482] T 2a67abb34f31418db9f6b9f3018ab368 P 2b55a36d7ff142c183080f141f6892bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:36:05.322310 9449 raft_consensus.cc:3037] T 2a67abb34f31418db9f6b9f3018ab368 P 2b55a36d7ff142c183080f141f6892bf [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:36:05.323566 9449 raft_consensus.cc:504] T 2a67abb34f31418db9f6b9f3018ab368 P 2b55a36d7ff142c183080f141f6892bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9cd141f38d4c21b0dd6fcbdfbb343f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 36365 } } peers { permanent_uuid: "5ccf4d68c3db4e05ba0ed000b51d552e" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 42487 } } peers { permanent_uuid: "2b55a36d7ff142c183080f141f6892bf" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43427 } }
I20260925 22:36:05.323905 9449 leader_election.cc:288] T 2a67abb34f31418db9f6b9f3018ab368 P 2b55a36d7ff142c183080f141f6892bf [CANDIDATE]: Term 1 election: Requested vote from peers 6a9cd141f38d4c21b0dd6fcbdfbb343f (127.6.246.193:36365), 5ccf4d68c3db4e05ba0ed000b51d552e (127.6.246.194:42487)
I20260925 22:36:05.324527 9169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a67abb34f31418db9f6b9f3018ab368" candidate_uuid: "2b55a36d7ff142c183080f141f6892bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ccf4d68c3db4e05ba0ed000b51d552e"
I20260925 22:36:05.324441 9029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a67abb34f31418db9f6b9f3018ab368" candidate_uuid: "2b55a36d7ff142c183080f141f6892bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a9cd141f38d4c21b0dd6fcbdfbb343f"
I20260925 22:36:05.324640 9169 raft_consensus.cc:3037] T 2a67abb34f31418db9f6b9f3018ab368 P 5ccf4d68c3db4e05ba0ed000b51d552e [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:36:05.324666 9029 raft_consensus.cc:3037] T 2a67abb34f31418db9f6b9f3018ab368 P 6a9cd141f38d4c21b0dd6fcbdfbb343f [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:36:05.325708 9169 raft_consensus.cc:2454] T 2a67abb34f31418db9f6b9f3018ab368 P 5ccf4d68c3db4e05ba0ed000b51d552e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b55a36d7ff142c183080f141f6892bf in term 1.
I20260925 22:36:05.325871 9029 raft_consensus.cc:2454] T 2a67abb34f31418db9f6b9f3018ab368 P 6a9cd141f38d4c21b0dd6fcbdfbb343f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b55a36d7ff142c183080f141f6892bf in term 1.
I20260925 22:36:05.326030 9241 leader_election.cc:302] T 2a67abb34f31418db9f6b9f3018ab368 P 2b55a36d7ff142c183080f141f6892bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b55a36d7ff142c183080f141f6892bf, 5ccf4d68c3db4e05ba0ed000b51d552e; no voters:
I20260925 22:36:05.326280 9449 raft_consensus.cc:2781] T 2a67abb34f31418db9f6b9f3018ab368 P 2b55a36d7ff142c183080f141f6892bf [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:36:05.326401 9449 raft_consensus.cc:686] T 2a67abb34f31418db9f6b9f3018ab368 P 2b55a36d7ff142c183080f141f6892bf [term 1 LEADER]: Becoming Leader. State: Replica: 2b55a36d7ff142c183080f141f6892bf, State: Running, Role: LEADER
I20260925 22:36:05.326604 9449 consensus_queue.cc:237] T 2a67abb34f31418db9f6b9f3018ab368 P 2b55a36d7ff142c183080f141f6892bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9cd141f38d4c21b0dd6fcbdfbb343f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 36365 } } peers { permanent_uuid: "5ccf4d68c3db4e05ba0ed000b51d552e" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 42487 } } peers { permanent_uuid: "2b55a36d7ff142c183080f141f6892bf" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43427 } }
I20260925 22:36:05.328711 8872 catalog_manager.cc:5495] T 2a67abb34f31418db9f6b9f3018ab368 P 2b55a36d7ff142c183080f141f6892bf reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b55a36d7ff142c183080f141f6892bf (127.6.246.195). New cstate: current_term: 1 leader_uuid: "2b55a36d7ff142c183080f141f6892bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9cd141f38d4c21b0dd6fcbdfbb343f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 36365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ccf4d68c3db4e05ba0ed000b51d552e" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 42487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b55a36d7ff142c183080f141f6892bf" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43427 } health_report { overall_health: HEALTHY } } }
I20260925 22:36:05.346009 9498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:05.342488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44098 (local address 127.6.246.195:43427)
0925 22:36:05.342823 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:05.342833 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:05.342845 (+ 12us) server_negotiation.cc:407] Connection header received
0925 22:36:05.342890 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:05.342893 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:05.342937 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:05.343032 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:36:05.343301 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:05.343969 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:05.344896 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:05.345140 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:05.345258 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:05.345505 (+ 247us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:36:05.345592 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:36:05.345670 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:05.345770 (+ 100us) server_negotiation.cc:299] Negotiation successful
0925 22:36:05.345829 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":132,"threads_started":1}
I20260925 22:36:05.355996 9501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:05.352869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.195:43427 (local address 127.6.246.195:49473)
0925 22:36:05.353208 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:05.353221 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:05.353339 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:05.353570 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:05.353573 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:05.353579 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 22:36:05.353667 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:05.353672 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:05.354444 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:05.354446 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:05.355047 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:05.355054 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:05.355442 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:05.355445 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:05.355503 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:05.355528 (+ 25us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 22:36:05.355756 (+ 228us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 22:36:05.355760 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 22:36:05.355809 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 22:36:05.355878 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":126,"threads_started":1}
I20260925 22:36:05.355996 9498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:05.353053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.195:49473 (local address 127.6.246.195:43427)
0925 22:36:05.353175 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:05.353179 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:05.353340 (+ 161us) server_negotiation.cc:407] Connection header received
0925 22:36:05.353427 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:05.353429 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:05.353480 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:05.353553 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:36:05.353778 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:05.354335 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:05.355142 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:05.355349 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:05.355427 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:05.355610 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:36:05.355679 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:36:05.355738 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:05.355816 (+ 78us) server_negotiation.cc:299] Negotiation successful
0925 22:36:05.355860 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 22:36:05.362008 9169 raft_consensus.cc:1261] T 2a67abb34f31418db9f6b9f3018ab368 P 5ccf4d68c3db4e05ba0ed000b51d552e [term 1 FOLLOWER]: Refusing update from remote peer 2b55a36d7ff142c183080f141f6892bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:36:05.362104 9029 raft_consensus.cc:1261] T 2a67abb34f31418db9f6b9f3018ab368 P 6a9cd141f38d4c21b0dd6fcbdfbb343f [term 1 FOLLOWER]: Refusing update from remote peer 2b55a36d7ff142c183080f141f6892bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:36:05.362550 9454 consensus_queue.cc:1035] T 2a67abb34f31418db9f6b9f3018ab368 P 2b55a36d7ff142c183080f141f6892bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a9cd141f38d4c21b0dd6fcbdfbb343f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 36365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:36:05.362686 9445 consensus_queue.cc:1035] T 2a67abb34f31418db9f6b9f3018ab368 P 2b55a36d7ff142c183080f141f6892bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ccf4d68c3db4e05ba0ed000b51d552e" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 42487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 22:36:05.377410 9447 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 22:36:06.933822 8872 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48486:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:36:06.934028 8872 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48486:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:36:06.936935 8872 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5f739135c3c04d60a053abf48962a868: Sending DeleteTablet for 3 replicas of tablet 2a67abb34f31418db9f6b9f3018ab368
I20260925 22:36:06.937644 9288 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a67abb34f31418db9f6b9f3018ab368 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:36:06 UTC) from {username='kudu', principal='kudu/127.6.246.254@KRBTEST.COM'} at 127.0.0.1:44078
I20260925 22:36:06.937726 9149 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a67abb34f31418db9f6b9f3018ab368 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:36:06 UTC) from {username='kudu', principal='kudu/127.6.246.254@KRBTEST.COM'} at 127.0.0.1:44572
I20260925 22:36:06.937721 9009 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a67abb34f31418db9f6b9f3018ab368 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:36:06 UTC) from {username='kudu', principal='kudu/127.6.246.254@KRBTEST.COM'} at 127.0.0.1:39684
I20260925 22:36:06.938048 9524 tablet_replica.cc:330] T 2a67abb34f31418db9f6b9f3018ab368 P 2b55a36d7ff142c183080f141f6892bf: stopping tablet replica
I20260925 22:36:06.938134 9526 tablet_replica.cc:330] T 2a67abb34f31418db9f6b9f3018ab368 P 6a9cd141f38d4c21b0dd6fcbdfbb343f: stopping tablet replica
I20260925 22:36:06.938163 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 8944
I20260925 22:36:06.938210 9524 raft_consensus.cc:2229] T 2a67abb34f31418db9f6b9f3018ab368 P 2b55a36d7ff142c183080f141f6892bf [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:36:06.939375 9525 tablet_replica.cc:330] T 2a67abb34f31418db9f6b9f3018ab368 P 5ccf4d68c3db4e05ba0ed000b51d552e: stopping tablet replica
I20260925 22:36:06.939615 9525 raft_consensus.cc:2229] T 2a67abb34f31418db9f6b9f3018ab368 P 5ccf4d68c3db4e05ba0ed000b51d552e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:36:06.939787 9525 raft_consensus.cc:2258] T 2a67abb34f31418db9f6b9f3018ab368 P 5ccf4d68c3db4e05ba0ed000b51d552e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:36:06.940917 9525 ts_tablet_manager.cc:1905] T 2a67abb34f31418db9f6b9f3018ab368 P 5ccf4d68c3db4e05ba0ed000b51d552e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:36:06.940966 9524 raft_consensus.cc:2258] T 2a67abb34f31418db9f6b9f3018ab368 P 2b55a36d7ff142c183080f141f6892bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:36:06.941800 9524 ts_tablet_manager.cc:1905] T 2a67abb34f31418db9f6b9f3018ab368 P 2b55a36d7ff142c183080f141f6892bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:36:06.944554 9525 ts_tablet_manager.cc:1918] T 2a67abb34f31418db9f6b9f3018ab368 P 5ccf4d68c3db4e05ba0ed000b51d552e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 22:36:06.944721 9525 log.cc:1196] T 2a67abb34f31418db9f6b9f3018ab368 P 5ccf4d68c3db4e05ba0ed000b51d552e: Deleting WAL directory at /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/minicluster-data/ts-1/wal/wals/2a67abb34f31418db9f6b9f3018ab368
I20260925 22:36:06.944988 9524 ts_tablet_manager.cc:1918] T 2a67abb34f31418db9f6b9f3018ab368 P 2b55a36d7ff142c183080f141f6892bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 22:36:06.945092 9524 log.cc:1196] T 2a67abb34f31418db9f6b9f3018ab368 P 2b55a36d7ff142c183080f141f6892bf: Deleting WAL directory at /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790375752797504-7131-0/minicluster-data/ts-2/wal/wals/2a67abb34f31418db9f6b9f3018ab368
I20260925 22:36:06.945120 9525 ts_tablet_manager.cc:1939] T 2a67abb34f31418db9f6b9f3018ab368 P 5ccf4d68c3db4e05ba0ed000b51d552e: Deleting consensus metadata
I20260925 22:36:06.945407 9524 ts_tablet_manager.cc:1939] T 2a67abb34f31418db9f6b9f3018ab368 P 2b55a36d7ff142c183080f141f6892bf: Deleting consensus metadata
I20260925 22:36:06.946154 8857 catalog_manager.cc:4841] TS 5ccf4d68c3db4e05ba0ed000b51d552e (127.6.246.194:42487): tablet 2a67abb34f31418db9f6b9f3018ab368 (table test-table [id=7929c1f8fa074e4b8755a6d083404f1e]) successfully deleted
I20260925 22:36:06.946177 8859 catalog_manager.cc:4841] TS 2b55a36d7ff142c183080f141f6892bf (127.6.246.195:43427): tablet 2a67abb34f31418db9f6b9f3018ab368 (table test-table [id=7929c1f8fa074e4b8755a6d083404f1e]) successfully deleted
W20260925 22:36:06.947499 8857 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.246.193:36365 (error 108)
I20260925 22:36:06.947808 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 9084
I20260925 22:36:06.949563 9528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:06.948955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.193:36365 (local address 127.0.0.1:39722)
0925 22:36:06.949302 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:06.949405 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.193:36365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":122,"threads_started":1}
W20260925 22:36:06.949717 8857 catalog_manager.cc:4584] TS 6a9cd141f38d4c21b0dd6fcbdfbb343f (127.6.246.193:36365): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2a67abb34f31418db9f6b9f3018ab368: Network error: Client connection negotiation failed: client connection to 127.6.246.193:36365: connect: Connection refused (error 111)
I20260925 22:36:06.957851 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 9224
I20260925 22:36:06.965859 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 8840
2026-09-25T22:36:06Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6587 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:36:07 dist-test-slave-wjdr krb5kdc[9531](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:36:07 dist-test-slave-wjdr krb5kdc[9531](info): set up 2 sockets
Sep 25 22:36:07 dist-test-slave-wjdr krb5kdc[9531](info): commencing operation
krb5kdc: starting...
W20260925 22:36:08.040526 7131 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Sep 25 22:36:08 dist-test-slave-wjdr krb5kdc[9531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375768, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:36:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:36:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.246.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab.
Entry for principal kudu/127.6.246.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab.
I20260925 22:36:08.164075 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:34469
--webserver_interface=127.6.246.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab
--principal=kudu/127.6.246.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:40985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.246.254:34469
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:36:08.273939 9544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:36:08.274183 9544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:36:08.274221 9544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:36:08.278175 9544 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:36:08.278241 9544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:36:08.278261 9544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:36:08.278278 9544 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:36:08.278294 9544 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:36:08.283653 9544 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:40985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.246.254:34469
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab
--principal=kudu/127.6.246.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:34469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.246.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9544
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:36:08.284758 9544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:36:08.285791 9544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:36:08.294476 9549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:08.294679 9552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:08.294495 9550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:08.295176 9544 server_base.cc:1034] running on GCE node
I20260925 22:36:08.295703 9544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:36:08.296819 9544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:36:08.297986 9544 hybrid_clock.cc:648] HybridClock initialized: now 1790375768297959 us; error 42 us; skew 500 ppm
Sep 25 22:36:08 dist-test-slave-wjdr krb5kdc[9531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375768, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:36:08.300848 9544 init.cc:377] Logged in from keytab as kudu/127.6.246.254@KRBTEST.COM (short username kudu)
I20260925 22:36:08.301702 9544 webserver.cc:460] Webserver started at http://127.6.246.254:38355/ using document root <none> and password file <none>
I20260925 22:36:08.302289 9544 fs_manager.cc:359] Metadata directory not provided
I20260925 22:36:08.302385 9544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:36:08.302618 9544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:36:08.304710 9544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/minicluster-data/master-0/data/instance:
uuid: "bf86e6b6478e4831b7e6182fda67c02e"
format_stamp: "Formatted at 2026-09-25 22:36:08 on dist-test-slave-wjdr"
server_key: "097e22964d646ccc7421aadb2c0dfad7"
server_key_iv: "cec91c93275ce29795be96b0b80fdbd2"
server_key_version: "encryptionkey@0"
I20260925 22:36:08.305274 9544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/minicluster-data/master-0/wal/instance:
uuid: "bf86e6b6478e4831b7e6182fda67c02e"
format_stamp: "Formatted at 2026-09-25 22:36:08 on dist-test-slave-wjdr"
server_key: "097e22964d646ccc7421aadb2c0dfad7"
server_key_iv: "cec91c93275ce29795be96b0b80fdbd2"
server_key_version: "encryptionkey@0"
I20260925 22:36:08.308915 9544 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:36:08.311239 9559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:08.312449 9544 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:36:08.312618 9544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/minicluster-data/master-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/minicluster-data/master-0/wal
uuid: "bf86e6b6478e4831b7e6182fda67c02e"
format_stamp: "Formatted at 2026-09-25 22:36:08 on dist-test-slave-wjdr"
server_key: "097e22964d646ccc7421aadb2c0dfad7"
server_key_iv: "cec91c93275ce29795be96b0b80fdbd2"
server_key_version: "encryptionkey@0"
I20260925 22:36:08.312716 9544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:36:08.331265 9544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:36:08.334750 9544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:36:08.334968 9544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:36:08.341790 9544 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.254:34469
I20260925 22:36:08.341857 9611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.254:34469 every 8 connection(s)
I20260925 22:36:08.342823 9544 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
I20260925 22:36:08.345551 9612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:36:08.350188 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 9544
I20260925 22:36:08.350327 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/minicluster-data/master-0/wal/instance
I20260925 22:36:08.350601 7131 external_mini_cluster.cc:1428] Setting key 235408bc674e46e65e0b80f10627d0fd
I20260925 22:36:08.351241 9612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf86e6b6478e4831b7e6182fda67c02e: Bootstrap starting.
I20260925 22:36:08.353890 9612 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf86e6b6478e4831b7e6182fda67c02e: Neither blocks nor log segments found. Creating new log.
I20260925 22:36:08.354750 9612 log.cc:824] T 00000000000000000000000000000000 P bf86e6b6478e4831b7e6182fda67c02e: Log is configured to *not* fsync() on all Append() calls
I20260925 22:36:08.356906 9612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf86e6b6478e4831b7e6182fda67c02e: No bootstrap required, opened a new log
Sep 25 22:36:08 dist-test-slave-wjdr krb5kdc[9531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375768, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
I20260925 22:36:08.360808 9612 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf86e6b6478e4831b7e6182fda67c02e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf86e6b6478e4831b7e6182fda67c02e" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 34469 } }
I20260925 22:36:08.361115 9612 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf86e6b6478e4831b7e6182fda67c02e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:36:08.361196 9612 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf86e6b6478e4831b7e6182fda67c02e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf86e6b6478e4831b7e6182fda67c02e, State: Initialized, Role: FOLLOWER
I20260925 22:36:08.361698 9612 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf86e6b6478e4831b7e6182fda67c02e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf86e6b6478e4831b7e6182fda67c02e" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 34469 } }
I20260925 22:36:08.361899 9612 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf86e6b6478e4831b7e6182fda67c02e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:36:08.361981 9612 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf86e6b6478e4831b7e6182fda67c02e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:36:08.362082 9612 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf86e6b6478e4831b7e6182fda67c02e [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:36:08.363206 9612 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf86e6b6478e4831b7e6182fda67c02e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf86e6b6478e4831b7e6182fda67c02e" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 34469 } }
I20260925 22:36:08.363569 9612 leader_election.cc:302] T 00000000000000000000000000000000 P bf86e6b6478e4831b7e6182fda67c02e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf86e6b6478e4831b7e6182fda67c02e; no voters:
I20260925 22:36:08.363910 9612 leader_election.cc:288] T 00000000000000000000000000000000 P bf86e6b6478e4831b7e6182fda67c02e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:36:08.364020 9617 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf86e6b6478e4831b7e6182fda67c02e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:36:08.364246 9617 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf86e6b6478e4831b7e6182fda67c02e [term 1 LEADER]: Becoming Leader. State: Replica: bf86e6b6478e4831b7e6182fda67c02e, State: Running, Role: LEADER
I20260925 22:36:08.364583 9617 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf86e6b6478e4831b7e6182fda67c02e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf86e6b6478e4831b7e6182fda67c02e" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 34469 } }
I20260925 22:36:08.365244 9612 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf86e6b6478e4831b7e6182fda67c02e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:36:08.366318 9619 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf86e6b6478e4831b7e6182fda67c02e [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf86e6b6478e4831b7e6182fda67c02e. Latest consensus state: current_term: 1 leader_uuid: "bf86e6b6478e4831b7e6182fda67c02e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf86e6b6478e4831b7e6182fda67c02e" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 34469 } } }
I20260925 22:36:08.366443 9615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:08.352156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45606 (local address 127.6.246.254:34469)
0925 22:36:08.352612 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:08.352634 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:08.352677 (+ 43us) server_negotiation.cc:407] Connection header received
0925 22:36:08.353480 (+ 803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:08.353501 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:08.353929 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:08.354302 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:08.355051 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:08.356406 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:08.357392 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:08.358038 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:08.358208 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:08.360687 (+ 2479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:08.360693 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:08.360710 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:08.360739 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:08.362802 (+ 2063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:08.363234 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:08.363240 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:08.363246 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:08.363397 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:08.363673 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:08.363676 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:08.363678 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:08.364196 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:08.364999 (+ 803us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:08.365144 (+ 145us) server_negotiation.cc:299] Negotiation successful
0925 22:36:08.365447 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":117,"threads_started":1}
I20260925 22:36:08.366470 9619 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf86e6b6478e4831b7e6182fda67c02e [sys.catalog]: This master's current role is: LEADER
I20260925 22:36:08.366314 9618 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf86e6b6478e4831b7e6182fda67c02e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf86e6b6478e4831b7e6182fda67c02e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf86e6b6478e4831b7e6182fda67c02e" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 34469 } } }
I20260925 22:36:08.366757 9618 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf86e6b6478e4831b7e6182fda67c02e [sys.catalog]: This master's current role is: LEADER
W20260925 22:36:08.369942 9632 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bf86e6b6478e4831b7e6182fda67c02e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:36:08.370033 9632 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:36:08.370097 9626 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:36:08.370741 9626 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:36:08.376356 9626 catalog_manager.cc:1347] Generated new cluster ID: 1d5979198820474db1da56fc59df8b82
I20260925 22:36:08.376426 9626 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:36:08.389187 9626 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:36:08.390147 9626 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:36:08.399626 9626 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf86e6b6478e4831b7e6182fda67c02e: Generated new TSK 0
I20260925 22:36:08.400411 9626 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.246.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.193.keytab.
Entry for principal kudu/127.6.246.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.193.keytab.
I20260925 22:36:08.439421 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.193:0
--local_ip_for_outbound_sockets=127.6.246.193
--webserver_interface=127.6.246.193
--webserver_port=0
--tserver_master_addrs=127.6.246.254:34469
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.193.keytab
--principal=kudu/127.6.246.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:40985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:36:08.550007 9638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:36:08.550307 9638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:36:08.550390 9638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:36:08.554427 9638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:36:08.554520 9638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:36:08.554615 9638 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.193
I20260925 22:36:08.559803 9638 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:40985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.193.keytab
--principal=kudu/127.6.246.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.246.193
--webserver_port=0
--tserver_master_addrs=127.6.246.254:34469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.193
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:36:08.560930 9638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:36:08.562011 9638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:36:08.572371 9644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:08.572311 9646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:08.572444 9638 server_base.cc:1034] running on GCE node
W20260925 22:36:08.572311 9643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:08.573032 9638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:36:08.573602 9638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:36:08.574810 9638 hybrid_clock.cc:648] HybridClock initialized: now 1790375768574756 us; error 28 us; skew 500 ppm
Sep 25 22:36:08 dist-test-slave-wjdr krb5kdc[9531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375768, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:36:08.577708 9638 init.cc:377] Logged in from keytab as kudu/127.6.246.193@KRBTEST.COM (short username kudu)
I20260925 22:36:08.578565 9638 webserver.cc:460] Webserver started at http://127.6.246.193:37879/ using document root <none> and password file <none>
I20260925 22:36:08.579111 9638 fs_manager.cc:359] Metadata directory not provided
I20260925 22:36:08.579203 9638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:36:08.579442 9638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:36:08.581521 9638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-0/data/instance:
uuid: "7e5c17280e8c4d8595c19941e5004897"
format_stamp: "Formatted at 2026-09-25 22:36:08 on dist-test-slave-wjdr"
server_key: "bcd9940c00b27958175beaad01a2e1ee"
server_key_iv: "706a15eaed8af1c58217dba996b7f0f2"
server_key_version: "encryptionkey@0"
I20260925 22:36:08.582080 9638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-0/wal/instance:
uuid: "7e5c17280e8c4d8595c19941e5004897"
format_stamp: "Formatted at 2026-09-25 22:36:08 on dist-test-slave-wjdr"
server_key: "bcd9940c00b27958175beaad01a2e1ee"
server_key_iv: "706a15eaed8af1c58217dba996b7f0f2"
server_key_version: "encryptionkey@0"
I20260925 22:36:08.585721 9638 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:36:08.588104 9653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:08.589349 9638 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:36:08.589494 9638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-0/wal
uuid: "7e5c17280e8c4d8595c19941e5004897"
format_stamp: "Formatted at 2026-09-25 22:36:08 on dist-test-slave-wjdr"
server_key: "bcd9940c00b27958175beaad01a2e1ee"
server_key_iv: "706a15eaed8af1c58217dba996b7f0f2"
server_key_version: "encryptionkey@0"
I20260925 22:36:08.589599 9638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:36:08.601572 9638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:36:08.604199 9638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:36:08.604401 9638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:36:08.604970 9638 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:36:08.605752 9638 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:36:08.605821 9638 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:08.605888 9638 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:36:08.605940 9638 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:08.615015 9638 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.193:33627
I20260925 22:36:08.615033 9766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.193:33627 every 8 connection(s)
I20260925 22:36:08.615970 9638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
Sep 25 22:36:08 dist-test-slave-wjdr krb5kdc[9531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375768, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.193@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
I20260925 22:36:08.626170 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 9638
I20260925 22:36:08.626300 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-0/wal/instance
I20260925 22:36:08.626680 7131 external_mini_cluster.cc:1428] Setting key 96f3be262a9853723d71c0872b88cbc4
I20260925 22:36:08.629882 9615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:08.617804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.193:49265 (local address 127.6.246.254:34469)
0925 22:36:08.617928 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:08.617932 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:08.618744 (+ 812us) server_negotiation.cc:407] Connection header received
0925 22:36:08.620127 (+ 1383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:08.620131 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:08.620189 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:08.620295 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:08.621497 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:08.622030 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:08.623163 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:08.623406 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:08.623516 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:08.626025 (+ 2509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:08.626035 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:08.626038 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:08.626063 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:08.627607 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:08.628182 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:08.628186 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:08.628188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:08.628281 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:08.628699 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:08.628701 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:08.628702 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:08.628894 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:08.628972 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:08.629604 (+ 632us) server_negotiation.cc:299] Negotiation successful
0925 22:36:08.629701 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 22:36:08.630863 9769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:08.618159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:34469 (local address 127.6.246.193:49265)
0925 22:36:08.618600 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:08.618642 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:08.619916 (+ 1274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:08.620509 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:08.620518 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:08.620893 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:08.621355 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:08.621368 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:08.622152 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:08.622155 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:08.623042 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:08.623051 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:08.623555 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:08.623559 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:08.623724 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:08.624282 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:08.624298 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:08.625846 (+ 1548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:36:08.627713 (+ 1867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:08.627717 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:08.627729 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:08.628082 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:08.628380 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:08.628383 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:08.628385 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:08.628611 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:08.629006 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:36:08.629015 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:36:08.629347 (+ 332us) client_negotiation.cc:769] Sending connection context
0925 22:36:08.629556 (+ 209us) client_negotiation.cc:240] Negotiation successful
0925 22:36:08.629914 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":112,"threads_started":1}
I20260925 22:36:08.632100 9767 heartbeater.cc:347] Connected to a master server at 127.6.246.254:34469
I20260925 22:36:08.632364 9767 heartbeater.cc:464] Registering TS with master...
I20260925 22:36:08.633023 9767 heartbeater.cc:511] Master 127.6.246.254:34469 requested a full tablet report, sending...
I20260925 22:36:08.634639 9576 ts_manager.cc:194] Registered new tserver with Master: 7e5c17280e8c4d8595c19941e5004897 (127.6.246.193:33627)
I20260925 22:36:08.635782 9576 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.246.193@KRBTEST.COM'} at 127.6.246.193:49265
WARNING: no policy specified for kudu/127.6.246.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.194.keytab.
Entry for principal kudu/127.6.246.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.194.keytab.
I20260925 22:36:08.655745 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.194:0
--local_ip_for_outbound_sockets=127.6.246.194
--webserver_interface=127.6.246.194
--webserver_port=0
--tserver_master_addrs=127.6.246.254:34469
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.194.keytab
--principal=kudu/127.6.246.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:40985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:36:08.766745 9772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:36:08.767038 9772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:36:08.767112 9772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:36:08.771143 9772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:36:08.771236 9772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:36:08.771333 9772 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.194
I20260925 22:36:08.776531 9772 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:40985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.194.keytab
--principal=kudu/127.6.246.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.246.194
--webserver_port=0
--tserver_master_addrs=127.6.246.254:34469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9772
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.194
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:36:08.777571 9772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:36:08.778581 9772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:36:08.788132 9780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:08.788136 9777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:08.788136 9778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:08.788369 9772 server_base.cc:1034] running on GCE node
I20260925 22:36:08.789044 9772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:36:08.789695 9772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:36:08.790867 9772 hybrid_clock.cc:648] HybridClock initialized: now 1790375768790850 us; error 29 us; skew 500 ppm
Sep 25 22:36:08 dist-test-slave-wjdr krb5kdc[9531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375768, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:36:08.794011 9772 init.cc:377] Logged in from keytab as kudu/127.6.246.194@KRBTEST.COM (short username kudu)
I20260925 22:36:08.794934 9772 webserver.cc:460] Webserver started at http://127.6.246.194:41021/ using document root <none> and password file <none>
I20260925 22:36:08.795439 9772 fs_manager.cc:359] Metadata directory not provided
I20260925 22:36:08.795498 9772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:36:08.795655 9772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:36:08.797660 9772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-1/data/instance:
uuid: "c66df97e9e7b4dca8b5466b43475a86f"
format_stamp: "Formatted at 2026-09-25 22:36:08 on dist-test-slave-wjdr"
server_key: "4c827bbe3b85bdd8c848c4e3d7713a05"
server_key_iv: "29fe6e602ba06e860c455b0f3019d0ab"
server_key_version: "encryptionkey@0"
I20260925 22:36:08.798164 9772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-1/wal/instance:
uuid: "c66df97e9e7b4dca8b5466b43475a86f"
format_stamp: "Formatted at 2026-09-25 22:36:08 on dist-test-slave-wjdr"
server_key: "4c827bbe3b85bdd8c848c4e3d7713a05"
server_key_iv: "29fe6e602ba06e860c455b0f3019d0ab"
server_key_version: "encryptionkey@0"
I20260925 22:36:08.801708 9772 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:36:08.804015 9787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:08.805244 9772 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:36:08.805378 9772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-1/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-1/wal
uuid: "c66df97e9e7b4dca8b5466b43475a86f"
format_stamp: "Formatted at 2026-09-25 22:36:08 on dist-test-slave-wjdr"
server_key: "4c827bbe3b85bdd8c848c4e3d7713a05"
server_key_iv: "29fe6e602ba06e860c455b0f3019d0ab"
server_key_version: "encryptionkey@0"
I20260925 22:36:08.805483 9772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:36:08.822402 9772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:36:08.825857 9772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:36:08.826071 9772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:36:08.826689 9772 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:36:08.827514 9772 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:36:08.827586 9772 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:08.827661 9772 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:36:08.827703 9772 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:08.836886 9772 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.194:42945
I20260925 22:36:08.836920 9900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.194:42945 every 8 connection(s)
I20260925 22:36:08.837984 9772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
I20260925 22:36:08.842329 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 9772
I20260925 22:36:08.842468 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-1/wal/instance
I20260925 22:36:08.842741 7131 external_mini_cluster.cc:1428] Setting key 66a8519411af97f2e262eec9fd5b102f
Sep 25 22:36:08 dist-test-slave-wjdr krb5kdc[9531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375768, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.194@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.246.195@KRBTEST.COM; defaulting to no policy
I20260925 22:36:08.852728 9615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:08.839762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.194:53135 (local address 127.6.246.254:34469)
0925 22:36:08.839939 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:08.839943 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:08.840806 (+ 863us) server_negotiation.cc:407] Connection header received
0925 22:36:08.842097 (+ 1291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:08.842100 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:08.842163 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:08.842260 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:08.843570 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:08.844262 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:08.846054 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:08.846439 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:08.846556 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:08.849246 (+ 2690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:08.849250 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:08.849254 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:08.849295 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:08.850678 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:08.851175 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:08.851181 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:08.851188 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:08.851260 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:08.851598 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:08.851601 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:08.851602 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:08.851778 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:08.851879 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:08.852418 (+ 539us) server_negotiation.cc:299] Negotiation successful
0925 22:36:08.852558 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260925 22:36:08.853590 9903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:08.840182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:34469 (local address 127.6.246.194:53135)
0925 22:36:08.840664 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:08.840706 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:08.841877 (+ 1171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:08.842416 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:08.842422 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:08.842872 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:08.843438 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:08.843454 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:08.844399 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:08.844402 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:08.845916 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:08.845932 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:08.846583 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:08.846585 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:08.846723 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:08.847327 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:08.847351 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:08.849082 (+ 1731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:36:08.850799 (+ 1717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:08.850806 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:08.850820 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:08.851069 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:08.851361 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:08.851366 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:08.851368 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:08.851498 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:08.851887 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:36:08.851897 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:36:08.852167 (+ 270us) client_negotiation.cc:769] Sending connection context
0925 22:36:08.852362 (+ 195us) client_negotiation.cc:240] Negotiation successful
0925 22:36:08.852698 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":106,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.195@KRBTEST.COM" created.
I20260925 22:36:08.854839 9901 heartbeater.cc:347] Connected to a master server at 127.6.246.254:34469
I20260925 22:36:08.855108 9901 heartbeater.cc:464] Registering TS with master...
I20260925 22:36:08.855687 9901 heartbeater.cc:511] Master 127.6.246.254:34469 requested a full tablet report, sending...
I20260925 22:36:08.856807 9576 ts_manager.cc:194] Registered new tserver with Master: c66df97e9e7b4dca8b5466b43475a86f (127.6.246.194:42945)
I20260925 22:36:08.857633 9576 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.246.194@KRBTEST.COM'} at 127.6.246.194:53135
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.195.keytab.
Entry for principal kudu/127.6.246.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.195.keytab.
I20260925 22:36:08.871165 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.195:0
--local_ip_for_outbound_sockets=127.6.246.195
--webserver_interface=127.6.246.195
--webserver_port=0
--tserver_master_addrs=127.6.246.254:34469
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.195.keytab
--principal=kudu/127.6.246.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:40985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:36:08.981401 9906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:36:08.981688 9906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:36:08.981765 9906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:36:08.985940 9906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:36:08.986023 9906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:36:08.986107 9906 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.195
I20260925 22:36:08.991266 9906 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:40985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.195.keytab
--principal=kudu/127.6.246.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.246.195
--webserver_port=0
--tserver_master_addrs=127.6.246.254:34469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9906
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.195
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:36:08.992578 9906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:36:08.993651 9906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:36:09.003196 9914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:09.003347 9911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:09.003199 9912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:09.003679 9906 server_base.cc:1034] running on GCE node
I20260925 22:36:09.004103 9906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:36:09.004786 9906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:36:09.005957 9906 hybrid_clock.cc:648] HybridClock initialized: now 1790375769005862 us; error 104 us; skew 500 ppm
Sep 25 22:36:09 dist-test-slave-wjdr krb5kdc[9531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375769, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:36:09.008823 9906 init.cc:377] Logged in from keytab as kudu/127.6.246.195@KRBTEST.COM (short username kudu)
I20260925 22:36:09.009753 9906 webserver.cc:460] Webserver started at http://127.6.246.195:37891/ using document root <none> and password file <none>
I20260925 22:36:09.010310 9906 fs_manager.cc:359] Metadata directory not provided
I20260925 22:36:09.010398 9906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:36:09.010632 9906 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:36:09.012684 9906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-2/data/instance:
uuid: "124cca46469e4d7e8acac1f0d249829d"
format_stamp: "Formatted at 2026-09-25 22:36:09 on dist-test-slave-wjdr"
server_key: "38c5cf2ac894f7604ab1ed59f2c5c78d"
server_key_iv: "e25e11936a78628d8bcd644b9f1fd623"
server_key_version: "encryptionkey@0"
I20260925 22:36:09.013240 9906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-2/wal/instance:
uuid: "124cca46469e4d7e8acac1f0d249829d"
format_stamp: "Formatted at 2026-09-25 22:36:09 on dist-test-slave-wjdr"
server_key: "38c5cf2ac894f7604ab1ed59f2c5c78d"
server_key_iv: "e25e11936a78628d8bcd644b9f1fd623"
server_key_version: "encryptionkey@0"
I20260925 22:36:09.016888 9906 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 22:36:09.019243 9921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:09.020388 9906 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:36:09.020555 9906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-2/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-2/wal
uuid: "124cca46469e4d7e8acac1f0d249829d"
format_stamp: "Formatted at 2026-09-25 22:36:09 on dist-test-slave-wjdr"
server_key: "38c5cf2ac894f7604ab1ed59f2c5c78d"
server_key_iv: "e25e11936a78628d8bcd644b9f1fd623"
server_key_version: "encryptionkey@0"
I20260925 22:36:09.020665 9906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:36:09.039227 9906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:36:09.042775 9906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:36:09.042985 9906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:36:09.043609 9906 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:36:09.044477 9906 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:36:09.044548 9906 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:09.044620 9906 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:36:09.044661 9906 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:09.053771 9906 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.195:40507
I20260925 22:36:09.053802 10034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.195:40507 every 8 connection(s)
I20260925 22:36:09.054675 9906 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
I20260925 22:36:09.057480 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 9906
I20260925 22:36:09.057610 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-2/wal/instance
I20260925 22:36:09.057945 7131 external_mini_cluster.cc:1428] Setting key 12efe500e2bedd4a609bc773d8efeda7
Sep 25 22:36:09 dist-test-slave-wjdr krb5kdc[9531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375769, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.195@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
I20260925 22:36:09.068210 9615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:09.056512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.195:45375 (local address 127.6.246.254:34469)
0925 22:36:09.056657 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:09.056661 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:09.057728 (+ 1067us) server_negotiation.cc:407] Connection header received
0925 22:36:09.058687 (+ 959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:09.058692 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:09.058745 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:09.058856 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:09.060081 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:09.060693 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:09.061846 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:09.062072 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:09.062182 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:09.064633 (+ 2451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:09.064636 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:09.064638 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:09.064664 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:09.066200 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:09.066705 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:09.066711 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:09.066713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:09.066804 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:09.067155 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:09.067160 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:09.067161 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:09.067343 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:09.067459 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:09.067931 (+ 472us) server_negotiation.cc:299] Negotiation successful
0925 22:36:09.068047 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 22:36:09.069092 10037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:09.056904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:34469 (local address 127.6.246.195:45375)
0925 22:36:09.057572 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:09.057634 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:09.058486 (+ 852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:09.059052 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:09.059059 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:09.059417 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:09.059936 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:09.059956 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:09.060834 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:09.060837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:09.061722 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:09.061731 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:09.062220 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:09.062224 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:09.062395 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:09.062977 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:09.062993 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:09.064415 (+ 1422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:36:09.066317 (+ 1902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:09.066322 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:09.066337 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:09.066599 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:09.066911 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:09.066914 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:09.066916 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:09.067058 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:09.067446 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:36:09.067453 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:36:09.067702 (+ 249us) client_negotiation.cc:769] Sending connection context
0925 22:36:09.067893 (+ 191us) client_negotiation.cc:240] Negotiation successful
0925 22:36:09.068228 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":112,"threads_started":1}
I20260925 22:36:09.070205 10035 heartbeater.cc:347] Connected to a master server at 127.6.246.254:34469
I20260925 22:36:09.070500 10035 heartbeater.cc:464] Registering TS with master...
I20260925 22:36:09.071113 10035 heartbeater.cc:511] Master 127.6.246.254:34469 requested a full tablet report, sending...
I20260925 22:36:09.072146 9576 ts_manager.cc:194] Registered new tserver with Master: 124cca46469e4d7e8acac1f0d249829d (127.6.246.195:40507)
I20260925 22:36:09.072786 9576 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.246.195@KRBTEST.COM'} at 127.6.246.195:45375
I20260925 22:36:09.083671 7131 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 22:36:09.096755 9615 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:36:09.095398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45618 (local address 127.6.246.254:34469)
0925 22:36:09.095518 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:09.095522 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:09.095706 (+ 184us) server_negotiation.cc:407] Connection header received
0925 22:36:09.095814 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:09.095817 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:09.095857 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:09.095935 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:09.096639 (+ 704us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45618: BlockingRecv error: recv got EOF from 127.0.0.1:45618 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 22:36:09.098397 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 9638
I20260925 22:36:09.103821 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 9772
I20260925 22:36:09.108961 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 9906
I20260925 22:36:09.114271 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 9544
2026-09-25T22:36:09Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2145 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790375752797504-7131-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:36:09 dist-test-slave-wjdr krb5kdc[10047](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:36:09 dist-test-slave-wjdr krb5kdc[10047](info): set up 2 sockets
Sep 25 22:36:09 dist-test-slave-wjdr krb5kdc[10047](info): commencing operation
krb5kdc: starting...
W20260925 22:36:10.152659 7131 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Sep 25 22:36:10 dist-test-slave-wjdr krb5kdc[10047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:36:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:36:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.246.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab.
Entry for principal kudu/127.6.246.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab.
I20260925 22:36:10.275590 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790375752797504-7131-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790375752797504-7131-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790375752797504-7131-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:40139
--webserver_interface=127.6.246.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab
--principal=kudu/127.6.246.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:36427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.246.254:40139
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:36:10.383250 10060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:36:10.383554 10060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:36:10.383627 10060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:36:10.387632 10060 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:36:10.387729 10060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:36:10.387773 10060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:36:10.387832 10060 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:36:10.387868 10060 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:36:10.393093 10060 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:36427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790375752797504-7131-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790375752797504-7131-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.246.254:40139
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab
--principal=kudu/127.6.246.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:40139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.246.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10060
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790375752797504-7131-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:36:10.394164 10060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:36:10.395171 10060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:36:10.403556 10066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:10.403601 10065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:10.403558 10068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:10.403721 10060 server_base.cc:1034] running on GCE node
I20260925 22:36:10.404521 10060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:36:10.405566 10060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:36:10.406749 10060 hybrid_clock.cc:648] HybridClock initialized: now 1790375770406731 us; error 31 us; skew 500 ppm
Sep 25 22:36:10 dist-test-slave-wjdr krb5kdc[10047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375770, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:36:10.414412 10060 init.cc:377] Logged in from keytab as kudu/127.6.246.254@KRBTEST.COM (short username kudu)
I20260925 22:36:10.415277 10060 webserver.cc:460] Webserver started at http://127.6.246.254:44867/ using document root <none> and password file <none>
I20260925 22:36:10.415789 10060 fs_manager.cc:359] Metadata directory not provided
I20260925 22:36:10.415846 10060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:36:10.416009 10060 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:36:10.418017 10060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790375752797504-7131-0/minicluster-data/master-0/data/instance:
uuid: "2e192996005548b2a53c77caa3c99ba0"
format_stamp: "Formatted at 2026-09-25 22:36:10 on dist-test-slave-wjdr"
server_key: "0a4257142319c3691a91b75ac28f708c"
server_key_iv: "66e1bb39d4d210a5a3b985ef29c30747"
server_key_version: "encryptionkey@0"
I20260925 22:36:10.418503 10060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790375752797504-7131-0/minicluster-data/master-0/wal/instance:
uuid: "2e192996005548b2a53c77caa3c99ba0"
format_stamp: "Formatted at 2026-09-25 22:36:10 on dist-test-slave-wjdr"
server_key: "0a4257142319c3691a91b75ac28f708c"
server_key_iv: "66e1bb39d4d210a5a3b985ef29c30747"
server_key_version: "encryptionkey@0"
I20260925 22:36:10.422075 10060 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260925 22:36:10.424312 10075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:10.425581 10060 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260925 22:36:10.425715 10060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790375752797504-7131-0/minicluster-data/master-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790375752797504-7131-0/minicluster-data/master-0/wal
uuid: "2e192996005548b2a53c77caa3c99ba0"
format_stamp: "Formatted at 2026-09-25 22:36:10 on dist-test-slave-wjdr"
server_key: "0a4257142319c3691a91b75ac28f708c"
server_key_iv: "66e1bb39d4d210a5a3b985ef29c30747"
server_key_version: "encryptionkey@0"
I20260925 22:36:10.425951 10060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790375752797504-7131-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790375752797504-7131-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790375752797504-7131-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:36:10.436430 10060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:36:10.439069 10060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:36:10.439253 10060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:36:10.445930 10060 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.254:40139
I20260925 22:36:10.446204 10127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.254:40139 every 8 connection(s)
I20260925 22:36:10.446830 10060 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
I20260925 22:36:10.449447 10128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:36:10.450948 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 10060
I20260925 22:36:10.451076 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790375752797504-7131-0/minicluster-data/master-0/wal/instance
I20260925 22:36:10.451406 7131 external_mini_cluster.cc:1428] Setting key 20687d3e0933e94330bb9d70e8a55aa6
I20260925 22:36:10.456312 10128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e192996005548b2a53c77caa3c99ba0: Bootstrap starting.
I20260925 22:36:10.458474 10128 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e192996005548b2a53c77caa3c99ba0: Neither blocks nor log segments found. Creating new log.
I20260925 22:36:10.459172 10128 log.cc:824] T 00000000000000000000000000000000 P 2e192996005548b2a53c77caa3c99ba0: Log is configured to *not* fsync() on all Append() calls
Sep 25 22:36:10 dist-test-slave-wjdr krb5kdc[10047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
I20260925 22:36:10.461313 10128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e192996005548b2a53c77caa3c99ba0: No bootstrap required, opened a new log
I20260925 22:36:10.465437 10131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:10.452766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59544 (local address 127.6.246.254:40139)
0925 22:36:10.453174 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:10.453190 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:10.453225 (+ 35us) server_negotiation.cc:407] Connection header received
0925 22:36:10.453870 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:10.453887 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:10.454206 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:10.454484 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:10.455375 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:10.456359 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:10.457261 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:10.457712 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:10.457892 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:10.460343 (+ 2451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:10.460349 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:10.460366 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:10.460395 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:10.462291 (+ 1896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:10.462724 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:10.462730 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:10.462736 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:10.462881 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:10.463187 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:10.463190 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:10.463192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:10.463753 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:10.463978 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:10.464168 (+ 190us) server_negotiation.cc:299] Negotiation successful
0925 22:36:10.464475 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":130,"threads_started":1}
I20260925 22:36:10.465798 10128 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e192996005548b2a53c77caa3c99ba0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e192996005548b2a53c77caa3c99ba0" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 40139 } }
I20260925 22:36:10.466005 10128 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e192996005548b2a53c77caa3c99ba0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:36:10.466063 10128 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e192996005548b2a53c77caa3c99ba0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e192996005548b2a53c77caa3c99ba0, State: Initialized, Role: FOLLOWER
I20260925 22:36:10.466562 10128 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e192996005548b2a53c77caa3c99ba0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e192996005548b2a53c77caa3c99ba0" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 40139 } }
I20260925 22:36:10.466717 10128 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e192996005548b2a53c77caa3c99ba0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:36:10.466781 10128 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e192996005548b2a53c77caa3c99ba0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:36:10.466879 10128 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e192996005548b2a53c77caa3c99ba0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:36:10.467989 10128 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e192996005548b2a53c77caa3c99ba0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e192996005548b2a53c77caa3c99ba0" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 40139 } }
I20260925 22:36:10.468353 10128 leader_election.cc:302] T 00000000000000000000000000000000 P 2e192996005548b2a53c77caa3c99ba0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e192996005548b2a53c77caa3c99ba0; no voters:
I20260925 22:36:10.468667 7131 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:36:10.468742 10128 leader_election.cc:288] T 00000000000000000000000000000000 P 2e192996005548b2a53c77caa3c99ba0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:36:10.468946 10133 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e192996005548b2a53c77caa3c99ba0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:36:10.469213 10133 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e192996005548b2a53c77caa3c99ba0 [term 1 LEADER]: Becoming Leader. State: Replica: 2e192996005548b2a53c77caa3c99ba0, State: Running, Role: LEADER
I20260925 22:36:10.469501 10133 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e192996005548b2a53c77caa3c99ba0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e192996005548b2a53c77caa3c99ba0" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 40139 } }
I20260925 22:36:10.470566 10133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e192996005548b2a53c77caa3c99ba0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e192996005548b2a53c77caa3c99ba0. Latest consensus state: current_term: 1 leader_uuid: "2e192996005548b2a53c77caa3c99ba0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e192996005548b2a53c77caa3c99ba0" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 40139 } } }
I20260925 22:36:10.470813 10133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e192996005548b2a53c77caa3c99ba0 [sys.catalog]: This master's current role is: LEADER
I20260925 22:36:10.470985 10134 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e192996005548b2a53c77caa3c99ba0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e192996005548b2a53c77caa3c99ba0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e192996005548b2a53c77caa3c99ba0" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 40139 } } }
I20260925 22:36:10.471089 10134 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e192996005548b2a53c77caa3c99ba0 [sys.catalog]: This master's current role is: LEADER
I20260925 22:36:10.471743 10128 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e192996005548b2a53c77caa3c99ba0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:36:10.471872 10138 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:36:10.475059 10138 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:36:10.481840 10138 catalog_manager.cc:1347] Generated new cluster ID: 9ee9062052834a8db5515d74ae88243d
I20260925 22:36:10.482008 10138 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:36:10.489231 10131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:10.483188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59558 (local address 127.6.246.254:40139)
0925 22:36:10.483318 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:10.483322 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:10.484759 (+ 1437us) server_negotiation.cc:407] Connection header received
0925 22:36:10.484919 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:10.484922 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:10.484973 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:10.485060 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:10.485576 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:10.486115 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:10.486965 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:10.487225 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:10.487338 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:10.488209 (+ 871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:10.488213 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:10.488216 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:36:10.488748 (+ 532us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:10.488778 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:36:10.488795 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:10.488914 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:10.489006 (+ 92us) server_negotiation.cc:299] Negotiation successful
0925 22:36:10.489068 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 22:36:10.495849 10138 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:36:10.497210 10138 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:36:10.501889 10138 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e192996005548b2a53c77caa3c99ba0: Generated new TSK 0
I20260925 22:36:10.502506 10138 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:36:10.511813 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 10060
2026-09-25T22:36:10Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1395 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:36:10 dist-test-slave-wjdr krb5kdc[10161](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:36:10 dist-test-slave-wjdr krb5kdc[10161](info): set up 2 sockets
Sep 25 22:36:10 dist-test-slave-wjdr krb5kdc[10161](info): commencing operation
krb5kdc: starting...
W20260925 22:36:11.611359 7131 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260925 22:36:11.817786 7131 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.206s user 0.000s sys 0.002s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260925 22:36:12.022468 7131 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.205s user 0.000s sys 0.002s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260925 22:36:12.255390 7131 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.233s user 0.000s sys 0.002s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/krb5kdc/test-user.keytab.
W20260925 22:36:12.881412 7131 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.626s user 0.000s sys 0.002s
Sep 25 22:36:12 dist-test-slave-wjdr krb5kdc[10161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:36:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:36:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.246.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab.
Entry for principal kudu/127.6.246.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab.
W20260925 22:36:14.254256 7131 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.6.246.254: real 1.351s user 0.001s sys 0.002s
I20260925 22:36:14.255621 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:41487
--webserver_interface=127.6.246.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab
--principal=kudu/127.6.246.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:44285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.246.254:41487
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:36:14.387588 10174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:36:14.387967 10174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:36:14.388048 10174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:36:14.392284 10174 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:36:14.392398 10174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:36:14.392433 10174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:36:14.392470 10174 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:36:14.392529 10174 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:36:14.398336 10174 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:44285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.246.254:41487
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab
--principal=kudu/127.6.246.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:41487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.246.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10174
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:36:14.399784 10174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:36:14.401181 10174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:36:14.411827 10182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:14.411841 10180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:14.411808 10179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:14.412178 10174 server_base.cc:1034] running on GCE node
I20260925 22:36:14.412904 10174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:36:14.414264 10174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:36:14.415437 10174 hybrid_clock.cc:648] HybridClock initialized: now 1790375774415416 us; error 42 us; skew 500 ppm
Sep 25 22:36:14 dist-test-slave-wjdr krb5kdc[10161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375774, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:36:14.418661 10174 init.cc:377] Logged in from keytab as kudu/127.6.246.254@KRBTEST.COM (short username kudu)
I20260925 22:36:14.419767 10174 webserver.cc:460] Webserver started at http://127.6.246.254:46679/ using document root <none> and password file <none>
I20260925 22:36:14.420501 10174 fs_manager.cc:359] Metadata directory not provided
I20260925 22:36:14.420580 10174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:36:14.420847 10174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:36:14.423414 10174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/minicluster-data/master-0/data/instance:
uuid: "fb0da56a43f3414290ffadc9732aa336"
format_stamp: "Formatted at 2026-09-25 22:36:14 on dist-test-slave-wjdr"
server_key: "e8e7b78272c8835982125a16dc169de4"
server_key_iv: "274ec7d433177f67aebb07a2c52ba470"
server_key_version: "encryptionkey@0"
I20260925 22:36:14.424070 10174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/minicluster-data/master-0/wal/instance:
uuid: "fb0da56a43f3414290ffadc9732aa336"
format_stamp: "Formatted at 2026-09-25 22:36:14 on dist-test-slave-wjdr"
server_key: "e8e7b78272c8835982125a16dc169de4"
server_key_iv: "274ec7d433177f67aebb07a2c52ba470"
server_key_version: "encryptionkey@0"
I20260925 22:36:14.428682 10174 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260925 22:36:14.431900 10189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:14.433588 10174 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:36:14.433763 10174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/minicluster-data/master-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/minicluster-data/master-0/wal
uuid: "fb0da56a43f3414290ffadc9732aa336"
format_stamp: "Formatted at 2026-09-25 22:36:14 on dist-test-slave-wjdr"
server_key: "e8e7b78272c8835982125a16dc169de4"
server_key_iv: "274ec7d433177f67aebb07a2c52ba470"
server_key_version: "encryptionkey@0"
I20260925 22:36:14.433911 10174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:36:14.456390 10174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:36:14.459846 10174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:36:14.460171 10174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:36:14.469293 10174 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.254:41487
I20260925 22:36:14.469684 10241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.254:41487 every 8 connection(s)
I20260925 22:36:14.470542 10174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
I20260925 22:36:14.473069 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 10174
I20260925 22:36:14.473222 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/minicluster-data/master-0/wal/instance
I20260925 22:36:14.473575 7131 external_mini_cluster.cc:1428] Setting key c2cd9da858e2a973a838703cf63cb7ce
I20260925 22:36:14.474490 10242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:36:14.481323 10242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb0da56a43f3414290ffadc9732aa336: Bootstrap starting.
Sep 25 22:36:14 dist-test-slave-wjdr krb5kdc[10161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
I20260925 22:36:14.483805 10242 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb0da56a43f3414290ffadc9732aa336: Neither blocks nor log segments found. Creating new log.
I20260925 22:36:14.484680 10242 log.cc:824] T 00000000000000000000000000000000 P fb0da56a43f3414290ffadc9732aa336: Log is configured to *not* fsync() on all Append() calls
I20260925 22:36:14.487012 10242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb0da56a43f3414290ffadc9732aa336: No bootstrap required, opened a new log
I20260925 22:36:14.489626 10245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:14.475261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33764 (local address 127.6.246.254:41487)
0925 22:36:14.475819 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:14.475838 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:14.475886 (+ 48us) server_negotiation.cc:407] Connection header received
0925 22:36:14.476767 (+ 881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:14.476791 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:14.477218 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:14.477658 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:14.478551 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:14.479710 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:14.480745 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:14.481277 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:14.481462 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:14.484241 (+ 2779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:14.484247 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:14.484264 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:14.484296 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:14.486392 (+ 2096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:14.486872 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:14.486878 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:14.486883 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:14.487005 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:14.487309 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:14.487314 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:14.487316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:14.487829 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:14.488063 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:14.488329 (+ 266us) server_negotiation.cc:299] Negotiation successful
0925 22:36:14.488710 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":188,"threads_started":1}
I20260925 22:36:14.491255 10242 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb0da56a43f3414290ffadc9732aa336 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0da56a43f3414290ffadc9732aa336" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 41487 } }
I20260925 22:36:14.491539 10242 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb0da56a43f3414290ffadc9732aa336 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:36:14.491623 10242 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb0da56a43f3414290ffadc9732aa336 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb0da56a43f3414290ffadc9732aa336, State: Initialized, Role: FOLLOWER
I20260925 22:36:14.492113 10242 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb0da56a43f3414290ffadc9732aa336 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0da56a43f3414290ffadc9732aa336" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 41487 } }
I20260925 22:36:14.492278 10242 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb0da56a43f3414290ffadc9732aa336 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:36:14.492352 10242 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb0da56a43f3414290ffadc9732aa336 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:36:14.492530 10242 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb0da56a43f3414290ffadc9732aa336 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:36:14.529701 10242 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb0da56a43f3414290ffadc9732aa336 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0da56a43f3414290ffadc9732aa336" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 41487 } }
I20260925 22:36:14.530326 10242 leader_election.cc:302] T 00000000000000000000000000000000 P fb0da56a43f3414290ffadc9732aa336 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb0da56a43f3414290ffadc9732aa336; no voters:
I20260925 22:36:14.530783 10242 leader_election.cc:288] T 00000000000000000000000000000000 P fb0da56a43f3414290ffadc9732aa336 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:36:14.531031 10247 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb0da56a43f3414290ffadc9732aa336 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:36:14.531368 10247 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb0da56a43f3414290ffadc9732aa336 [term 1 LEADER]: Becoming Leader. State: Replica: fb0da56a43f3414290ffadc9732aa336, State: Running, Role: LEADER
I20260925 22:36:14.531868 10247 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb0da56a43f3414290ffadc9732aa336 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0da56a43f3414290ffadc9732aa336" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 41487 } }
I20260925 22:36:14.532600 10242 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb0da56a43f3414290ffadc9732aa336 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:36:14.534103 10249 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb0da56a43f3414290ffadc9732aa336 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb0da56a43f3414290ffadc9732aa336. Latest consensus state: current_term: 1 leader_uuid: "fb0da56a43f3414290ffadc9732aa336" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0da56a43f3414290ffadc9732aa336" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 41487 } } }
I20260925 22:36:14.534373 10249 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb0da56a43f3414290ffadc9732aa336 [sys.catalog]: This master's current role is: LEADER
I20260925 22:36:14.534408 10248 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb0da56a43f3414290ffadc9732aa336 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb0da56a43f3414290ffadc9732aa336" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0da56a43f3414290ffadc9732aa336" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 41487 } } }
I20260925 22:36:14.534613 10248 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb0da56a43f3414290ffadc9732aa336 [sys.catalog]: This master's current role is: LEADER
I20260925 22:36:14.535019 10256 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:36:14.539209 10256 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:36:14.546330 10256 catalog_manager.cc:1347] Generated new cluster ID: 0bebcb17992048118b4b047a373c68eb
I20260925 22:36:14.546418 10256 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:36:14.553594 10256 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:36:14.554651 10256 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:36:14.564558 10256 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb0da56a43f3414290ffadc9732aa336: Generated new TSK 0
I20260925 22:36:14.565431 10256 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.246.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.193.keytab.
Entry for principal kudu/127.6.246.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.193.keytab.
W20260925 22:36:15.405133 7131 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.6.246.193: real 0.826s user 0.000s sys 0.004s
I20260925 22:36:15.405833 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.193:0
--local_ip_for_outbound_sockets=127.6.246.193
--webserver_interface=127.6.246.193
--webserver_port=0
--tserver_master_addrs=127.6.246.254:41487
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.193.keytab
--principal=kudu/127.6.246.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:44285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:36:15.533787 10268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:36:15.534189 10268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:36:15.534302 10268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:36:15.538687 10268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:36:15.538746 10268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:36:15.538844 10268 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.193
I20260925 22:36:15.544307 10268 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:44285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.193.keytab
--principal=kudu/127.6.246.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.246.193
--webserver_port=0
--tserver_master_addrs=127.6.246.254:41487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10268
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.193
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:36:15.545629 10268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:36:15.546885 10268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:36:15.557971 10273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:15.557960 10276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:15.557956 10274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:15.558270 10268 server_base.cc:1034] running on GCE node
I20260925 22:36:15.558917 10268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:36:15.559689 10268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:36:15.560876 10268 hybrid_clock.cc:648] HybridClock initialized: now 1790375775560854 us; error 40 us; skew 500 ppm
Sep 25 22:36:15 dist-test-slave-wjdr krb5kdc[10161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375775, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:36:15.568974 10268 init.cc:377] Logged in from keytab as kudu/127.6.246.193@KRBTEST.COM (short username kudu)
I20260925 22:36:15.570037 10268 webserver.cc:460] Webserver started at http://127.6.246.193:42553/ using document root <none> and password file <none>
I20260925 22:36:15.570710 10268 fs_manager.cc:359] Metadata directory not provided
I20260925 22:36:15.570778 10268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:36:15.571009 10268 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:36:15.573673 10268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-0/data/instance:
uuid: "571dbbde01da419d8703a0129fa8244b"
format_stamp: "Formatted at 2026-09-25 22:36:15 on dist-test-slave-wjdr"
server_key: "377830bca8eacb788284641908c6a02f"
server_key_iv: "9faa4d23596ae77b20be38cf33525a23"
server_key_version: "encryptionkey@0"
I20260925 22:36:15.574313 10268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-0/wal/instance:
uuid: "571dbbde01da419d8703a0129fa8244b"
format_stamp: "Formatted at 2026-09-25 22:36:15 on dist-test-slave-wjdr"
server_key: "377830bca8eacb788284641908c6a02f"
server_key_iv: "9faa4d23596ae77b20be38cf33525a23"
server_key_version: "encryptionkey@0"
I20260925 22:36:15.578872 10268 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:36:15.581869 10283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:15.583534 10268 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 22:36:15.583678 10268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-0/wal
uuid: "571dbbde01da419d8703a0129fa8244b"
format_stamp: "Formatted at 2026-09-25 22:36:15 on dist-test-slave-wjdr"
server_key: "377830bca8eacb788284641908c6a02f"
server_key_iv: "9faa4d23596ae77b20be38cf33525a23"
server_key_version: "encryptionkey@0"
I20260925 22:36:15.583806 10268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:36:15.601589 10268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:36:15.605508 10268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:36:15.605762 10268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:36:15.606513 10268 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:36:15.607533 10268 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:36:15.607589 10268 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:15.607676 10268 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:36:15.607707 10268 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:15.618146 10268 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.193:45031
I20260925 22:36:15.618170 10396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.193:45031 every 8 connection(s)
I20260925 22:36:15.619240 10268 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
I20260925 22:36:15.624073 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 10268
I20260925 22:36:15.624217 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-0/wal/instance
I20260925 22:36:15.624629 7131 external_mini_cluster.cc:1428] Setting key 1d521a9682c0e152a8ae4e3322ec8a05
Sep 25 22:36:15 dist-test-slave-wjdr krb5kdc[10161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375775, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.193@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.246.194@KRBTEST.COM; defaulting to no policy
I20260925 22:36:15.635733 10399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:15.621499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.193:39941 (local address 127.6.246.254:41487)
0925 22:36:15.621791 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:15.621794 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:15.622618 (+ 824us) server_negotiation.cc:407] Connection header received
0925 22:36:15.624198 (+ 1580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:15.624201 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:15.624285 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:15.624418 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:15.625860 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:15.626470 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:15.627720 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:15.628178 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:15.628303 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:15.631172 (+ 2869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:15.631179 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:15.631186 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:15.631221 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:15.633161 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:15.633755 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:15.633761 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:15.633765 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:15.633886 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:15.634333 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:15.634339 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:15.634343 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:15.634629 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:15.634741 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:15.635321 (+ 580us) server_negotiation.cc:299] Negotiation successful
0925 22:36:15.635473 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":87,"threads_started":1}
I20260925 22:36:15.636811 10400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:15.621874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:41487 (local address 127.6.246.193:39941)
0925 22:36:15.622426 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:15.622485 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:15.623963 (+ 1478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:15.624653 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:15.624661 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:15.625123 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:15.625702 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:15.625717 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:15.626608 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:15.626614 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:15.627588 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:15.627597 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:15.628342 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:15.628345 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:15.628588 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:15.629266 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:15.629284 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:15.630987 (+ 1703us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:36:15.633281 (+ 2294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:15.633286 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:15.633301 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:15.633628 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:15.634006 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:15.634010 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:15.634012 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:15.634174 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:15.634753 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:36:15.634759 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:36:15.635054 (+ 295us) client_negotiation.cc:769] Sending connection context
0925 22:36:15.635232 (+ 178us) client_negotiation.cc:240] Negotiation successful
0925 22:36:15.635592 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":148,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.194@KRBTEST.COM" created.
I20260925 22:36:15.638368 10397 heartbeater.cc:347] Connected to a master server at 127.6.246.254:41487
I20260925 22:36:15.638672 10397 heartbeater.cc:464] Registering TS with master...
I20260925 22:36:15.639307 10397 heartbeater.cc:511] Master 127.6.246.254:41487 requested a full tablet report, sending...
I20260925 22:36:15.641153 10206 ts_manager.cc:194] Registered new tserver with Master: 571dbbde01da419d8703a0129fa8244b (127.6.246.193:45031)
I20260925 22:36:15.642952 10206 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.246.193@KRBTEST.COM'} at 127.6.246.193:39941
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.194.keytab.
Entry for principal kudu/127.6.246.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.194.keytab.
I20260925 22:36:15.655308 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.194:0
--local_ip_for_outbound_sockets=127.6.246.194
--webserver_interface=127.6.246.194
--webserver_port=0
--tserver_master_addrs=127.6.246.254:41487
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.194.keytab
--principal=kudu/127.6.246.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:44285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:36:15.786441 10403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:36:15.786798 10403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:36:15.786849 10403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:36:15.791241 10403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:36:15.791311 10403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:36:15.791375 10403 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.194
I20260925 22:36:15.796967 10403 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:44285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.194.keytab
--principal=kudu/127.6.246.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.246.194
--webserver_port=0
--tserver_master_addrs=127.6.246.254:41487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10403
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.194
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:36:15.798266 10403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:36:15.799541 10403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:36:15.810709 10411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:15.810796 10408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:15.810727 10409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:15.810976 10403 server_base.cc:1034] running on GCE node
I20260925 22:36:15.811625 10403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:36:15.812346 10403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:36:15.813540 10403 hybrid_clock.cc:648] HybridClock initialized: now 1790375775813506 us; error 47 us; skew 500 ppm
Sep 25 22:36:15 dist-test-slave-wjdr krb5kdc[10161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375775, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:36:15.816725 10403 init.cc:377] Logged in from keytab as kudu/127.6.246.194@KRBTEST.COM (short username kudu)
I20260925 22:36:15.817739 10403 webserver.cc:460] Webserver started at http://127.6.246.194:35977/ using document root <none> and password file <none>
I20260925 22:36:15.818426 10403 fs_manager.cc:359] Metadata directory not provided
I20260925 22:36:15.818519 10403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:36:15.818783 10403 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:36:15.821354 10403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-1/data/instance:
uuid: "15c60335e01d409c9249e33cfd45fed5"
format_stamp: "Formatted at 2026-09-25 22:36:15 on dist-test-slave-wjdr"
server_key: "99e7fa7de3e6203c1919c72a43b3dc7f"
server_key_iv: "c7195458679054fadba7b42b13bdf8d9"
server_key_version: "encryptionkey@0"
I20260925 22:36:15.822029 10403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-1/wal/instance:
uuid: "15c60335e01d409c9249e33cfd45fed5"
format_stamp: "Formatted at 2026-09-25 22:36:15 on dist-test-slave-wjdr"
server_key: "99e7fa7de3e6203c1919c72a43b3dc7f"
server_key_iv: "c7195458679054fadba7b42b13bdf8d9"
server_key_version: "encryptionkey@0"
I20260925 22:36:15.826462 10403 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:36:15.829335 10418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:15.830724 10403 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 22:36:15.830883 10403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-1/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-1/wal
uuid: "15c60335e01d409c9249e33cfd45fed5"
format_stamp: "Formatted at 2026-09-25 22:36:15 on dist-test-slave-wjdr"
server_key: "99e7fa7de3e6203c1919c72a43b3dc7f"
server_key_iv: "c7195458679054fadba7b42b13bdf8d9"
server_key_version: "encryptionkey@0"
I20260925 22:36:15.831008 10403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:36:15.855760 10403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:36:15.859445 10403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:36:15.859723 10403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:36:15.860560 10403 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:36:15.861658 10403 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:36:15.861743 10403 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:15.861833 10403 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:36:15.861898 10403 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:15.873165 10403 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.194:37335
I20260925 22:36:15.873188 10531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.194:37335 every 8 connection(s)
I20260925 22:36:15.874325 10403 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
I20260925 22:36:15.883271 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 10403
I20260925 22:36:15.883452 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-1/wal/instance
I20260925 22:36:15.883906 7131 external_mini_cluster.cc:1428] Setting key b3cdd057c9cc0a163333ed006999f655
Sep 25 22:36:15 dist-test-slave-wjdr krb5kdc[10161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375775, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.194@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
I20260925 22:36:15.891511 10399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:15.876715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.194:54311 (local address 127.6.246.254:41487)
0925 22:36:15.876910 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:15.876916 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:15.878036 (+ 1120us) server_negotiation.cc:407] Connection header received
0925 22:36:15.879544 (+ 1508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:15.879546 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:15.879607 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:15.879703 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:15.881137 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:15.881794 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:15.883070 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:15.883521 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:15.883653 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:15.887199 (+ 3546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:15.887203 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:15.887205 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:15.887228 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:15.889126 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:15.889650 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:15.889655 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:15.889657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:15.889755 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:15.890128 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:15.890133 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:15.890138 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:15.890414 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:15.890517 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:15.891079 (+ 562us) server_negotiation.cc:299] Negotiation successful
0925 22:36:15.891261 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 22:36:15.892448 10534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:15.877255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:41487 (local address 127.6.246.194:54311)
0925 22:36:15.877848 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:15.877899 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:15.879291 (+ 1392us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:15.879896 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:15.879904 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:15.880298 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:15.880978 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:15.880993 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:15.881917 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:15.881920 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:15.882878 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:15.882887 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:15.883696 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:15.883698 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:15.883883 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:15.884699 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:15.884728 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:15.887011 (+ 2283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:36:15.889238 (+ 2227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:15.889243 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:15.889258 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:15.889526 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:15.889852 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:15.889854 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:15.889856 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:15.889996 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:15.890519 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:36:15.890524 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:36:15.890782 (+ 258us) client_negotiation.cc:769] Sending connection context
0925 22:36:15.890971 (+ 189us) client_negotiation.cc:240] Negotiation successful
0925 22:36:15.891294 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":155,"threads_started":1}
I20260925 22:36:15.893886 10532 heartbeater.cc:347] Connected to a master server at 127.6.246.254:41487
I20260925 22:36:15.894207 10532 heartbeater.cc:464] Registering TS with master...
I20260925 22:36:15.894775 10532 heartbeater.cc:511] Master 127.6.246.254:41487 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.6.246.195@KRBTEST.COM; defaulting to no policy
I20260925 22:36:15.896015 10206 ts_manager.cc:194] Registered new tserver with Master: 15c60335e01d409c9249e33cfd45fed5 (127.6.246.194:37335)
I20260925 22:36:15.896689 10206 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.246.194@KRBTEST.COM'} at 127.6.246.194:54311
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.195.keytab.
Entry for principal kudu/127.6.246.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.195.keytab.
I20260925 22:36:15.915835 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.195:0
--local_ip_for_outbound_sockets=127.6.246.195
--webserver_interface=127.6.246.195
--webserver_port=0
--tserver_master_addrs=127.6.246.254:41487
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.195.keytab
--principal=kudu/127.6.246.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:44285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:36:16.040282 10537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:36:16.040587 10537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:36:16.040663 10537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:36:16.045475 10537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:36:16.045570 10537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:36:16.045667 10537 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.195
I20260925 22:36:16.053002 10537 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:44285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.195.keytab
--principal=kudu/127.6.246.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.246.195
--webserver_port=0
--tserver_master_addrs=127.6.246.254:41487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10537
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.195
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:36:16.054186 10537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:36:16.055267 10537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:36:16.065183 10545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:16.065161 10543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:16.065219 10542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:16.065449 10537 server_base.cc:1034] running on GCE node
I20260925 22:36:16.065907 10537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:36:16.066519 10537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:36:16.067693 10537 hybrid_clock.cc:648] HybridClock initialized: now 1790375776067653 us; error 55 us; skew 500 ppm
Sep 25 22:36:16 dist-test-slave-wjdr krb5kdc[10161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375776, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:36:16.070494 10537 init.cc:377] Logged in from keytab as kudu/127.6.246.195@KRBTEST.COM (short username kudu)
I20260925 22:36:16.071326 10537 webserver.cc:460] Webserver started at http://127.6.246.195:45835/ using document root <none> and password file <none>
I20260925 22:36:16.071877 10537 fs_manager.cc:359] Metadata directory not provided
I20260925 22:36:16.071935 10537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:36:16.072108 10537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:36:16.074244 10537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-2/data/instance:
uuid: "60af82f7d02a45c9b3deb26f8688dc15"
format_stamp: "Formatted at 2026-09-25 22:36:16 on dist-test-slave-wjdr"
server_key: "c726f1c5c4db44bc1b3bf40171781f59"
server_key_iv: "feb4a860af6f35f3d6a41be70a358457"
server_key_version: "encryptionkey@0"
I20260925 22:36:16.074761 10537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-2/wal/instance:
uuid: "60af82f7d02a45c9b3deb26f8688dc15"
format_stamp: "Formatted at 2026-09-25 22:36:16 on dist-test-slave-wjdr"
server_key: "c726f1c5c4db44bc1b3bf40171781f59"
server_key_iv: "feb4a860af6f35f3d6a41be70a358457"
server_key_version: "encryptionkey@0"
I20260925 22:36:16.078475 10537 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 22:36:16.080912 10552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:16.082188 10537 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:36:16.082288 10537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-2/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-2/wal
uuid: "60af82f7d02a45c9b3deb26f8688dc15"
format_stamp: "Formatted at 2026-09-25 22:36:16 on dist-test-slave-wjdr"
server_key: "c726f1c5c4db44bc1b3bf40171781f59"
server_key_iv: "feb4a860af6f35f3d6a41be70a358457"
server_key_version: "encryptionkey@0"
I20260925 22:36:16.082422 10537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:36:16.096808 10537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:36:16.099992 10537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:36:16.100142 10537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:36:16.100673 10537 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:36:16.101444 10537 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:36:16.101491 10537 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:16.101529 10537 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:36:16.101545 10537 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:16.110065 10537 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.195:46803
I20260925 22:36:16.110327 10665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.195:46803 every 8 connection(s)
I20260925 22:36:16.111078 10537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
I20260925 22:36:16.113169 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 10537
I20260925 22:36:16.113329 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790375752797504-7131-0/minicluster-data/ts-2/wal/instance
I20260925 22:36:16.113686 7131 external_mini_cluster.cc:1428] Setting key ed0cdbefeef16e963111de2b5b523573
Sep 25 22:36:16 dist-test-slave-wjdr krb5kdc[10161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375776, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.195@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
I20260925 22:36:16.124797 10399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:16.112923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.195:56429 (local address 127.6.246.254:41487)
0925 22:36:16.113106 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:16.113111 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:16.114069 (+ 958us) server_negotiation.cc:407] Connection header received
0925 22:36:16.115782 (+ 1713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:16.115785 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:16.115854 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:16.115947 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:16.117184 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:16.117735 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:16.118872 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:16.119083 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:16.119190 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:16.121603 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:16.121607 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:16.121609 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:16.121629 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:16.122938 (+ 1309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:16.123391 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:16.123393 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:16.123394 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:16.123463 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:16.123737 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:16.123739 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:16.123740 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:16.123920 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:16.123997 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:16.124569 (+ 572us) server_negotiation.cc:299] Negotiation successful
0925 22:36:16.124670 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 22:36:16.125702 10668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:16.113307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:41487 (local address 127.6.246.195:56429)
0925 22:36:16.113901 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:16.113961 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:16.115545 (+ 1584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:16.116120 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:16.116126 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:16.116547 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:16.117052 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:16.117065 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:16.117856 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:16.117859 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:16.118757 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:16.118766 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:16.119241 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:16.119244 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:16.119401 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:16.119928 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:16.119943 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:16.121430 (+ 1487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:36:16.123041 (+ 1611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:16.123045 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:16.123055 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:16.123294 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:16.123540 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:16.123542 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:16.123543 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:16.123663 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:16.124034 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:36:16.124039 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:36:16.124255 (+ 216us) client_negotiation.cc:769] Sending connection context
0925 22:36:16.124518 (+ 263us) client_negotiation.cc:240] Negotiation successful
0925 22:36:16.124785 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":113,"threads_started":1}
I20260925 22:36:16.126863 10666 heartbeater.cc:347] Connected to a master server at 127.6.246.254:41487
I20260925 22:36:16.127152 10666 heartbeater.cc:464] Registering TS with master...
I20260925 22:36:16.127701 10666 heartbeater.cc:511] Master 127.6.246.254:41487 requested a full tablet report, sending...
I20260925 22:36:16.128648 10206 ts_manager.cc:194] Registered new tserver with Master: 60af82f7d02a45c9b3deb26f8688dc15 (127.6.246.195:46803)
I20260925 22:36:16.129184 10206 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.246.195@KRBTEST.COM'} at 127.6.246.195:56429
I20260925 22:36:16.139282 7131 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 22:36:16 dist-test-slave-wjdr krb5kdc[10161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375776, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 25 22:36:16 dist-test-slave-wjdr krb5kdc[10161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375776, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
I20260925 22:36:16.165256 10399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:16.156260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33772 (local address 127.6.246.254:41487)
0925 22:36:16.156417 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:16.156423 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:16.156596 (+ 173us) server_negotiation.cc:407] Connection header received
0925 22:36:16.156772 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:16.156775 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:16.156822 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:16.156903 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:16.157620 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:16.158147 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:16.159280 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:16.159498 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:16.159605 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:16.162190 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:16.162194 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:16.162196 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:16.162218 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:16.163504 (+ 1286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:16.163921 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:16.163923 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:16.163924 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:16.163994 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:16.164267 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:16.164269 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:16.164270 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:16.164556 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:16.164709 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:16.164986 (+ 277us) server_negotiation.cc:299] Negotiation successful
0925 22:36:16.165101 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260925 22:36:16.165657 10206 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:33772
I20260925 22:36:16.166636 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 10268
I20260925 22:36:16.172673 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 10403
I20260925 22:36:16.178539 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 10537
I20260925 22:36:16.183832 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 10174
2026-09-25T22:36:16Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (5674 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:36:16 dist-test-slave-wjdr krb5kdc[10678](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:36:16 dist-test-slave-wjdr krb5kdc[10678](info): set up 2 sockets
Sep 25 22:36:16 dist-test-slave-wjdr krb5kdc[10678](info): commencing operation
krb5kdc: starting...
W20260925 22:36:17.220686 7131 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Sep 25 22:36:17 dist-test-slave-wjdr krb5kdc[10678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375777, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:36:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:36:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.246.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab.
Entry for principal kudu/127.6.246.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab.
I20260925 22:36:17.339771 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:45291
--webserver_interface=127.6.246.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab
--principal=kudu/127.6.246.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:45827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.246.254:45291
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:36:17.461674 10691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:36:17.461961 10691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:36:17.462033 10691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:36:17.465986 10691 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:36:17.466076 10691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:36:17.466115 10691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:36:17.466174 10691 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:36:17.466214 10691 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:36:17.471360 10691 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:45827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.246.254:45291
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab
--principal=kudu/127.6.246.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:45291
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.246.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:36:17.472448 10691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:36:17.473551 10691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:36:17.481760 10696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:17.481998 10691 server_base.cc:1034] running on GCE node
W20260925 22:36:17.481765 10697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:17.481771 10699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:17.482625 10691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:36:17.483668 10691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:36:17.484869 10691 hybrid_clock.cc:648] HybridClock initialized: now 1790375777484850 us; error 30 us; skew 500 ppm
Sep 25 22:36:17 dist-test-slave-wjdr krb5kdc[10678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375777, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:36:17.488008 10691 init.cc:377] Logged in from keytab as kudu/127.6.246.254@KRBTEST.COM (short username kudu)
I20260925 22:36:17.489007 10691 webserver.cc:460] Webserver started at http://127.6.246.254:36843/ using document root <none> and password file <none>
I20260925 22:36:17.489545 10691 fs_manager.cc:359] Metadata directory not provided
I20260925 22:36:17.489605 10691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:36:17.489765 10691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:36:17.491719 10691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/minicluster-data/master-0/data/instance:
uuid: "e01ae19e8206424fafc00d1a70de5e34"
format_stamp: "Formatted at 2026-09-25 22:36:17 on dist-test-slave-wjdr"
server_key: "ca21d8f94adddfe023b2b4d3e1f43963"
server_key_iv: "aa849fe1478501a1aa02598e93dbf1da"
server_key_version: "encryptionkey@0"
I20260925 22:36:17.492209 10691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/minicluster-data/master-0/wal/instance:
uuid: "e01ae19e8206424fafc00d1a70de5e34"
format_stamp: "Formatted at 2026-09-25 22:36:17 on dist-test-slave-wjdr"
server_key: "ca21d8f94adddfe023b2b4d3e1f43963"
server_key_iv: "aa849fe1478501a1aa02598e93dbf1da"
server_key_version: "encryptionkey@0"
I20260925 22:36:17.495947 10691 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 22:36:17.498447 10706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:17.499612 10691 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:36:17.499737 10691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/minicluster-data/master-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/minicluster-data/master-0/wal
uuid: "e01ae19e8206424fafc00d1a70de5e34"
format_stamp: "Formatted at 2026-09-25 22:36:17 on dist-test-slave-wjdr"
server_key: "ca21d8f94adddfe023b2b4d3e1f43963"
server_key_iv: "aa849fe1478501a1aa02598e93dbf1da"
server_key_version: "encryptionkey@0"
I20260925 22:36:17.499837 10691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:36:17.512012 10691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:36:17.514838 10691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:36:17.515038 10691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:36:17.522256 10691 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.254:45291
I20260925 22:36:17.522284 10758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.254:45291 every 8 connection(s)
I20260925 22:36:17.523155 10691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
I20260925 22:36:17.525884 10759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:36:17.527011 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 10691
I20260925 22:36:17.527109 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/minicluster-data/master-0/wal/instance
I20260925 22:36:17.527417 7131 external_mini_cluster.cc:1428] Setting key e00bf2d360f7f5ca09989ef9cbde1349
I20260925 22:36:17.531826 10759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e01ae19e8206424fafc00d1a70de5e34: Bootstrap starting.
I20260925 22:36:17.534129 10759 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e01ae19e8206424fafc00d1a70de5e34: Neither blocks nor log segments found. Creating new log.
I20260925 22:36:17.534866 10759 log.cc:824] T 00000000000000000000000000000000 P e01ae19e8206424fafc00d1a70de5e34: Log is configured to *not* fsync() on all Append() calls
Sep 25 22:36:17 dist-test-slave-wjdr krb5kdc[10678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375777, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
I20260925 22:36:17.536944 10759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e01ae19e8206424fafc00d1a70de5e34: No bootstrap required, opened a new log
I20260925 22:36:17.540584 10759 raft_consensus.cc:348] T 00000000000000000000000000000000 P e01ae19e8206424fafc00d1a70de5e34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e01ae19e8206424fafc00d1a70de5e34" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 45291 } }
I20260925 22:36:17.540838 10759 raft_consensus.cc:374] T 00000000000000000000000000000000 P e01ae19e8206424fafc00d1a70de5e34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:36:17.540926 10759 raft_consensus.cc:729] T 00000000000000000000000000000000 P e01ae19e8206424fafc00d1a70de5e34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e01ae19e8206424fafc00d1a70de5e34, State: Initialized, Role: FOLLOWER
I20260925 22:36:17.541425 10759 consensus_queue.cc:260] T 00000000000000000000000000000000 P e01ae19e8206424fafc00d1a70de5e34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e01ae19e8206424fafc00d1a70de5e34" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 45291 } }
I20260925 22:36:17.541492 10762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:17.528850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43356 (local address 127.6.246.254:45291)
0925 22:36:17.529315 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:17.529330 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:17.529365 (+ 35us) server_negotiation.cc:407] Connection header received
0925 22:36:17.529994 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:17.530010 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:17.530326 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:17.530686 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:17.531492 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:17.532569 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:17.533414 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:17.533901 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:17.534072 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:17.536847 (+ 2775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:17.536852 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:17.536866 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:17.536893 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:17.538565 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:17.539012 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:17.539017 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:17.539021 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:17.539125 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:17.539411 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:17.539414 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:17.539416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:17.539789 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:17.540006 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:17.540279 (+ 273us) server_negotiation.cc:299] Negotiation successful
0925 22:36:17.540618 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":136,"threads_started":1}
I20260925 22:36:17.541594 10759 raft_consensus.cc:388] T 00000000000000000000000000000000 P e01ae19e8206424fafc00d1a70de5e34 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:36:17.541666 10759 raft_consensus.cc:482] T 00000000000000000000000000000000 P e01ae19e8206424fafc00d1a70de5e34 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:36:17.541769 10759 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e01ae19e8206424fafc00d1a70de5e34 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:36:17.542865 10759 raft_consensus.cc:504] T 00000000000000000000000000000000 P e01ae19e8206424fafc00d1a70de5e34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e01ae19e8206424fafc00d1a70de5e34" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 45291 } }
I20260925 22:36:17.543197 10759 leader_election.cc:302] T 00000000000000000000000000000000 P e01ae19e8206424fafc00d1a70de5e34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e01ae19e8206424fafc00d1a70de5e34; no voters:
I20260925 22:36:17.543522 10759 leader_election.cc:288] T 00000000000000000000000000000000 P e01ae19e8206424fafc00d1a70de5e34 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:36:17.543704 10764 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e01ae19e8206424fafc00d1a70de5e34 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:36:17.543958 10764 raft_consensus.cc:686] T 00000000000000000000000000000000 P e01ae19e8206424fafc00d1a70de5e34 [term 1 LEADER]: Becoming Leader. State: Replica: e01ae19e8206424fafc00d1a70de5e34, State: Running, Role: LEADER
I20260925 22:36:17.544310 10764 consensus_queue.cc:237] T 00000000000000000000000000000000 P e01ae19e8206424fafc00d1a70de5e34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e01ae19e8206424fafc00d1a70de5e34" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 45291 } }
I20260925 22:36:17.544826 10759 sys_catalog.cc:564] T 00000000000000000000000000000000 P e01ae19e8206424fafc00d1a70de5e34 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:36:17.546057 10765 sys_catalog.cc:455] T 00000000000000000000000000000000 P e01ae19e8206424fafc00d1a70de5e34 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e01ae19e8206424fafc00d1a70de5e34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e01ae19e8206424fafc00d1a70de5e34" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 45291 } } }
I20260925 22:36:17.546193 10765 sys_catalog.cc:458] T 00000000000000000000000000000000 P e01ae19e8206424fafc00d1a70de5e34 [sys.catalog]: This master's current role is: LEADER
I20260925 22:36:17.546500 10766 sys_catalog.cc:455] T 00000000000000000000000000000000 P e01ae19e8206424fafc00d1a70de5e34 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e01ae19e8206424fafc00d1a70de5e34. Latest consensus state: current_term: 1 leader_uuid: "e01ae19e8206424fafc00d1a70de5e34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e01ae19e8206424fafc00d1a70de5e34" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 45291 } } }
I20260925 22:36:17.546609 10766 sys_catalog.cc:458] T 00000000000000000000000000000000 P e01ae19e8206424fafc00d1a70de5e34 [sys.catalog]: This master's current role is: LEADER
I20260925 22:36:17.546587 10773 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:36:17.550122 10773 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:36:17.555691 10773 catalog_manager.cc:1347] Generated new cluster ID: 2c93dff9b27643469ac155dc95c19d1b
I20260925 22:36:17.555758 10773 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:36:17.561460 10773 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:36:17.562351 10773 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:36:17.570459 10773 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e01ae19e8206424fafc00d1a70de5e34: Generated new TSK 0
I20260925 22:36:17.571187 10773 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.246.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.193.keytab.
Entry for principal kudu/127.6.246.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.193.keytab.
I20260925 22:36:17.601557 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.193:0
--local_ip_for_outbound_sockets=127.6.246.193
--webserver_interface=127.6.246.193
--webserver_port=0
--tserver_master_addrs=127.6.246.254:45291
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.193.keytab
--principal=kudu/127.6.246.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:45827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:36:17.710557 10785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:36:17.710778 10785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:36:17.710819 10785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:36:17.714687 10785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:36:17.714732 10785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:36:17.714787 10785 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.193
I20260925 22:36:17.720645 10785 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:45827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.193.keytab
--principal=kudu/127.6.246.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.246.193
--webserver_port=0
--tserver_master_addrs=127.6.246.254:45291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10785
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.193
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:36:17.721776 10785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:36:17.722759 10785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:36:17.732092 10790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:17.732105 10793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:17.732110 10791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:17.732290 10785 server_base.cc:1034] running on GCE node
I20260925 22:36:17.732781 10785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:36:17.733388 10785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:36:17.734570 10785 hybrid_clock.cc:648] HybridClock initialized: now 1790375777734563 us; error 30 us; skew 500 ppm
Sep 25 22:36:17 dist-test-slave-wjdr krb5kdc[10678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375777, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:36:17.737267 10785 init.cc:377] Logged in from keytab as kudu/127.6.246.193@KRBTEST.COM (short username kudu)
I20260925 22:36:17.738008 10785 webserver.cc:460] Webserver started at http://127.6.246.193:32973/ using document root <none> and password file <none>
I20260925 22:36:17.738492 10785 fs_manager.cc:359] Metadata directory not provided
I20260925 22:36:17.738547 10785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:36:17.738698 10785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:36:17.740621 10785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/minicluster-data/ts-0/data/instance:
uuid: "658b8f25746342278f04c303eabda386"
format_stamp: "Formatted at 2026-09-25 22:36:17 on dist-test-slave-wjdr"
server_key: "c8feb42bb285677f1831074eb9cd24a7"
server_key_iv: "86d18fe6853b203ec0ec4ccadb1f4bda"
server_key_version: "encryptionkey@0"
I20260925 22:36:17.741114 10785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/minicluster-data/ts-0/wal/instance:
uuid: "658b8f25746342278f04c303eabda386"
format_stamp: "Formatted at 2026-09-25 22:36:17 on dist-test-slave-wjdr"
server_key: "c8feb42bb285677f1831074eb9cd24a7"
server_key_iv: "86d18fe6853b203ec0ec4ccadb1f4bda"
server_key_version: "encryptionkey@0"
I20260925 22:36:17.744618 10785 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:36:17.747095 10800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:17.748277 10785 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:36:17.748365 10785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/minicluster-data/ts-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/minicluster-data/ts-0/wal
uuid: "658b8f25746342278f04c303eabda386"
format_stamp: "Formatted at 2026-09-25 22:36:17 on dist-test-slave-wjdr"
server_key: "c8feb42bb285677f1831074eb9cd24a7"
server_key_iv: "86d18fe6853b203ec0ec4ccadb1f4bda"
server_key_version: "encryptionkey@0"
I20260925 22:36:17.748554 10785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:36:17.768749 10785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:36:17.771755 10785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:36:17.771963 10785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:36:17.772568 10785 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:36:17.773391 10785 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:36:17.773437 10785 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:17.773476 10785 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:36:17.773533 10785 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:17.782910 10785 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.193:43865
I20260925 22:36:17.782938 10913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.193:43865 every 8 connection(s)
I20260925 22:36:17.783772 10785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
I20260925 22:36:17.787595 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 10785
I20260925 22:36:17.787722 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/minicluster-data/ts-0/wal/instance
I20260925 22:36:17.788005 7131 external_mini_cluster.cc:1428] Setting key e2d49e0198af4d55321b2d6493e70e8d
Sep 25 22:36:17 dist-test-slave-wjdr krb5kdc[10678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375777, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.193@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.246.194@KRBTEST.COM; defaulting to no policy
I20260925 22:36:17.799237 10762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:17.785517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.193:60065 (local address 127.6.246.254:45291)
0925 22:36:17.785660 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:17.785664 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:17.786544 (+ 880us) server_negotiation.cc:407] Connection header received
0925 22:36:17.788099 (+ 1555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:17.788103 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:17.788167 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:17.788265 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:17.789912 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:17.790737 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:17.792537 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:17.792767 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:17.792846 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:17.795655 (+ 2809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:17.795658 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:17.795660 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:17.795681 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:17.797154 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:17.797701 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:17.797704 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:17.797705 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:17.797775 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:17.798108 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:17.798113 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:17.798115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:17.798288 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:17.798392 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:17.798952 (+ 560us) server_negotiation.cc:299] Negotiation successful
0925 22:36:17.799068 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.194@KRBTEST.COM" created.
I20260925 22:36:17.800132 10916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:17.785922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:45291 (local address 127.6.246.193:60065)
0925 22:36:17.786397 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:17.786441 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:17.787893 (+ 1452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:17.788451 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:17.788461 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:17.789080 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:17.789766 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:17.789787 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:17.790860 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:17.790863 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:17.792383 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:17.792397 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:17.792911 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:17.792918 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:17.793159 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:17.793823 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:17.793852 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:17.795484 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:36:17.797282 (+ 1798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:17.797287 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:17.797301 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:17.797565 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:17.797866 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:17.797871 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:17.797873 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:17.798012 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:17.798397 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:36:17.798406 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:36:17.798698 (+ 292us) client_negotiation.cc:769] Sending connection context
0925 22:36:17.798905 (+ 207us) client_negotiation.cc:240] Negotiation successful
0925 22:36:17.799206 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":101,"threads_started":1}
I20260925 22:36:17.801470 10914 heartbeater.cc:347] Connected to a master server at 127.6.246.254:45291
I20260925 22:36:17.801752 10914 heartbeater.cc:464] Registering TS with master...
I20260925 22:36:17.802367 10914 heartbeater.cc:511] Master 127.6.246.254:45291 requested a full tablet report, sending...
I20260925 22:36:17.804026 10723 ts_manager.cc:194] Registered new tserver with Master: 658b8f25746342278f04c303eabda386 (127.6.246.193:43865)
I20260925 22:36:17.805224 10723 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.246.193@KRBTEST.COM'} at 127.6.246.193:60065
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.194.keytab.
Entry for principal kudu/127.6.246.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.194.keytab.
I20260925 22:36:17.815934 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.194:0
--local_ip_for_outbound_sockets=127.6.246.194
--webserver_interface=127.6.246.194
--webserver_port=0
--tserver_master_addrs=127.6.246.254:45291
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.194.keytab
--principal=kudu/127.6.246.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:45827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:36:17.922032 10919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:36:17.922307 10919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:36:17.922381 10919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:36:17.926357 10919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:36:17.926434 10919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:36:17.926539 10919 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.194
I20260925 22:36:17.931823 10919 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:45827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.194.keytab
--principal=kudu/127.6.246.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.246.194
--webserver_port=0
--tserver_master_addrs=127.6.246.254:45291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10919
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.194
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:36:17.932807 10919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:36:17.933727 10919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:36:17.942938 10927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:17.943008 10924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:17.942996 10925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:17.943104 10919 server_base.cc:1034] running on GCE node
I20260925 22:36:17.943558 10919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:36:17.944145 10919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:36:17.945307 10919 hybrid_clock.cc:648] HybridClock initialized: now 1790375777945288 us; error 29 us; skew 500 ppm
Sep 25 22:36:17 dist-test-slave-wjdr krb5kdc[10678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375777, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:36:17.948304 10919 init.cc:377] Logged in from keytab as kudu/127.6.246.194@KRBTEST.COM (short username kudu)
I20260925 22:36:17.949163 10919 webserver.cc:460] Webserver started at http://127.6.246.194:37421/ using document root <none> and password file <none>
I20260925 22:36:17.949682 10919 fs_manager.cc:359] Metadata directory not provided
I20260925 22:36:17.949781 10919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:36:17.949975 10919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:36:17.952033 10919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/minicluster-data/ts-1/data/instance:
uuid: "bc1cb14357e74f2bbae79e630b074b4e"
format_stamp: "Formatted at 2026-09-25 22:36:17 on dist-test-slave-wjdr"
server_key: "b522dc8e728f7d4cf212f1b882613ac3"
server_key_iv: "804c2c53347202363bcfe696d984b471"
server_key_version: "encryptionkey@0"
I20260925 22:36:17.952687 10919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/minicluster-data/ts-1/wal/instance:
uuid: "bc1cb14357e74f2bbae79e630b074b4e"
format_stamp: "Formatted at 2026-09-25 22:36:17 on dist-test-slave-wjdr"
server_key: "b522dc8e728f7d4cf212f1b882613ac3"
server_key_iv: "804c2c53347202363bcfe696d984b471"
server_key_version: "encryptionkey@0"
I20260925 22:36:17.956208 10919 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260925 22:36:17.958616 10934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:17.959796 10919 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:36:17.959915 10919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/minicluster-data/ts-1/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/minicluster-data/ts-1/wal
uuid: "bc1cb14357e74f2bbae79e630b074b4e"
format_stamp: "Formatted at 2026-09-25 22:36:17 on dist-test-slave-wjdr"
server_key: "b522dc8e728f7d4cf212f1b882613ac3"
server_key_iv: "804c2c53347202363bcfe696d984b471"
server_key_version: "encryptionkey@0"
I20260925 22:36:17.960012 10919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:36:17.992034 10919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:36:17.995422 10919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:36:17.995630 10919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:36:17.996234 10919 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:36:17.997103 10919 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:36:17.997150 10919 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:17.997216 10919 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:36:17.997263 10919 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:18.006275 10919 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.194:40215
I20260925 22:36:18.006318 11047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.194:40215 every 8 connection(s)
I20260925 22:36:18.007158 10919 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
I20260925 22:36:18.011621 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 10919
I20260925 22:36:18.011751 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/minicluster-data/ts-1/wal/instance
I20260925 22:36:18.012077 7131 external_mini_cluster.cc:1428] Setting key 9f08f6a458a55766d838db92a84b10e9
Sep 25 22:36:18 dist-test-slave-wjdr krb5kdc[10678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375777, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.194@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.246.195@KRBTEST.COM; defaulting to no policy
I20260925 22:36:18.021662 10762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:18.008967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.194:46775 (local address 127.6.246.254:45291)
0925 22:36:18.009108 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:18.009113 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:18.010032 (+ 919us) server_negotiation.cc:407] Connection header received
0925 22:36:18.011406 (+ 1374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:18.011411 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:18.011469 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:18.011577 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:18.012981 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:18.013621 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:18.014805 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:18.015088 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:18.015172 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:18.018094 (+ 2922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:18.018099 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:18.018107 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:18.018130 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:18.019574 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:18.020086 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:18.020092 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:18.020093 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:18.020175 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:18.020550 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:18.020553 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:18.020555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:18.020732 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:18.020840 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:18.021376 (+ 536us) server_negotiation.cc:299] Negotiation successful
0925 22:36:18.021498 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 22:36:18.022456 11050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:18.009425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:45291 (local address 127.6.246.194:46775)
0925 22:36:18.009877 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:18.009921 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:18.011148 (+ 1227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:18.011793 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:18.011802 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:18.012226 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:18.012834 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:18.012849 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:18.013768 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:18.013772 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:18.014642 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:18.014650 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:18.015241 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:18.015254 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:18.015481 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:18.016084 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:18.016108 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:18.017931 (+ 1823us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:36:18.019695 (+ 1764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:18.019701 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:18.019715 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:18.019981 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:18.020269 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:18.020274 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:18.020276 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:18.020412 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:18.020837 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:36:18.020843 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:36:18.021125 (+ 282us) client_negotiation.cc:769] Sending connection context
0925 22:36:18.021309 (+ 184us) client_negotiation.cc:240] Negotiation successful
0925 22:36:18.021613 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":109,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.195@KRBTEST.COM" created.
I20260925 22:36:18.023761 11048 heartbeater.cc:347] Connected to a master server at 127.6.246.254:45291
I20260925 22:36:18.024005 11048 heartbeater.cc:464] Registering TS with master...
I20260925 22:36:18.024529 11048 heartbeater.cc:511] Master 127.6.246.254:45291 requested a full tablet report, sending...
I20260925 22:36:18.025597 10723 ts_manager.cc:194] Registered new tserver with Master: bc1cb14357e74f2bbae79e630b074b4e (127.6.246.194:40215)
I20260925 22:36:18.026158 10723 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.246.194@KRBTEST.COM'} at 127.6.246.194:46775
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.195.keytab.
Entry for principal kudu/127.6.246.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.195.keytab.
I20260925 22:36:18.037405 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.195:0
--local_ip_for_outbound_sockets=127.6.246.195
--webserver_interface=127.6.246.195
--webserver_port=0
--tserver_master_addrs=127.6.246.254:45291
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.195.keytab
--principal=kudu/127.6.246.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:45827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:36:18.146749 11053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:36:18.146976 11053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:36:18.147068 11053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:36:18.151048 11053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:36:18.151095 11053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:36:18.151201 11053 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.195
I20260925 22:36:18.156327 11053 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:45827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.195.keytab
--principal=kudu/127.6.246.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.246.195
--webserver_port=0
--tserver_master_addrs=127.6.246.254:45291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11053
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.195
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:36:18.157276 11053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:36:18.158231 11053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:36:18.167613 11058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:18.167646 11061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:18.167677 11059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:18.167909 11053 server_base.cc:1034] running on GCE node
I20260925 22:36:18.168365 11053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:36:18.169049 11053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:36:18.170253 11053 hybrid_clock.cc:648] HybridClock initialized: now 1790375778170219 us; error 56 us; skew 500 ppm
Sep 25 22:36:18 dist-test-slave-wjdr krb5kdc[10678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375778, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:36:18.173151 11053 init.cc:377] Logged in from keytab as kudu/127.6.246.195@KRBTEST.COM (short username kudu)
I20260925 22:36:18.174098 11053 webserver.cc:460] Webserver started at http://127.6.246.195:34115/ using document root <none> and password file <none>
I20260925 22:36:18.174678 11053 fs_manager.cc:359] Metadata directory not provided
I20260925 22:36:18.174773 11053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:36:18.175082 11053 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:36:18.177274 11053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/minicluster-data/ts-2/data/instance:
uuid: "6beb3ac3068f4d62806da33abc08e69f"
format_stamp: "Formatted at 2026-09-25 22:36:18 on dist-test-slave-wjdr"
server_key: "f52ccd802e4f737f94a92359b3cb113d"
server_key_iv: "84c01254af66dbc25a1e475b4d5cd2b0"
server_key_version: "encryptionkey@0"
I20260925 22:36:18.177901 11053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/minicluster-data/ts-2/wal/instance:
uuid: "6beb3ac3068f4d62806da33abc08e69f"
format_stamp: "Formatted at 2026-09-25 22:36:18 on dist-test-slave-wjdr"
server_key: "f52ccd802e4f737f94a92359b3cb113d"
server_key_iv: "84c01254af66dbc25a1e475b4d5cd2b0"
server_key_version: "encryptionkey@0"
I20260925 22:36:18.181836 11053 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:36:18.184291 11068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:18.185516 11053 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:36:18.185657 11053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/minicluster-data/ts-2/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/minicluster-data/ts-2/wal
uuid: "6beb3ac3068f4d62806da33abc08e69f"
format_stamp: "Formatted at 2026-09-25 22:36:18 on dist-test-slave-wjdr"
server_key: "f52ccd802e4f737f94a92359b3cb113d"
server_key_iv: "84c01254af66dbc25a1e475b4d5cd2b0"
server_key_version: "encryptionkey@0"
I20260925 22:36:18.185770 11053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:36:18.200381 11053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:36:18.203267 11053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:36:18.203459 11053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:36:18.204101 11053 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:36:18.204993 11053 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:36:18.205077 11053 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:18.205157 11053 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:36:18.205217 11053 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:18.214432 11053 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.195:42863
I20260925 22:36:18.214475 11181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.195:42863 every 8 connection(s)
I20260925 22:36:18.215310 11053 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
I20260925 22:36:18.222834 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 11053
I20260925 22:36:18.222975 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790375752797504-7131-0/minicluster-data/ts-2/wal/instance
I20260925 22:36:18.223359 7131 external_mini_cluster.cc:1428] Setting key df06e7aa04655955be83097399e13b17
Sep 25 22:36:18 dist-test-slave-wjdr krb5kdc[10678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375778, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.195@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
I20260925 22:36:18.228750 10762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:18.216919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.195:44669 (local address 127.6.246.254:45291)
0925 22:36:18.217050 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:18.217053 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:18.217863 (+ 810us) server_negotiation.cc:407] Connection header received
0925 22:36:18.219146 (+ 1283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:18.219149 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:18.219201 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:18.219285 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:18.220392 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:18.221093 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:18.222159 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:18.222398 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:18.222487 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:18.225434 (+ 2947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:18.225438 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:18.225441 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:18.225470 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:18.226805 (+ 1335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:18.227285 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:18.227288 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:18.227289 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:18.227369 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:18.227660 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:18.227663 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:18.227664 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:18.227860 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:18.227938 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:18.228520 (+ 582us) server_negotiation.cc:299] Negotiation successful
0925 22:36:18.228627 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 22:36:18.229600 11184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:18.217282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:45291 (local address 127.6.246.195:44669)
0925 22:36:18.217700 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:18.217737 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:18.218942 (+ 1205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:18.219434 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:18.219441 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:18.219768 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:18.220223 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:18.220236 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:18.221201 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:18.221204 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:18.222041 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:18.222050 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:18.222526 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:18.222528 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:18.222676 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:18.223297 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:18.223320 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:18.225261 (+ 1941us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:36:18.226916 (+ 1655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:18.226921 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:18.226932 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:18.227174 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:18.227456 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:18.227458 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:18.227460 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:18.227577 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:18.227993 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:36:18.227998 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:36:18.228210 (+ 212us) client_negotiation.cc:769] Sending connection context
0925 22:36:18.228383 (+ 173us) client_negotiation.cc:240] Negotiation successful
0925 22:36:18.228727 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":117,"threads_started":1}
I20260925 22:36:18.230669 11182 heartbeater.cc:347] Connected to a master server at 127.6.246.254:45291
I20260925 22:36:18.230970 11182 heartbeater.cc:464] Registering TS with master...
I20260925 22:36:18.231449 11182 heartbeater.cc:511] Master 127.6.246.254:45291 requested a full tablet report, sending...
I20260925 22:36:18.232445 10723 ts_manager.cc:194] Registered new tserver with Master: 6beb3ac3068f4d62806da33abc08e69f (127.6.246.195:42863)
I20260925 22:36:18.233076 10723 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.246.195@KRBTEST.COM'} at 127.6.246.195:44669
I20260925 22:36:18.237146 7131 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 22:36:18 dist-test-slave-wjdr krb5kdc[10678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375778, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 22:36:18 dist-test-slave-wjdr krb5kdc[10678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375778, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.246.193@KRBTEST.COM
I20260925 22:36:18.263058 11189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:18.253648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38524 (local address 127.6.246.193:43865)
0925 22:36:18.253956 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:18.253970 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:18.253997 (+ 27us) server_negotiation.cc:407] Connection header received
0925 22:36:18.254108 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:18.254114 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:18.254292 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:18.254439 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:18.255099 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:18.255857 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:18.256753 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:18.257110 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:18.257230 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:18.259705 (+ 2475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:18.259713 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:18.259725 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:18.259747 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:18.261247 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:18.261680 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:18.261685 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:18.261687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:18.261763 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:18.262033 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:18.262038 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:18.262040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:18.262268 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:18.262504 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:18.262661 (+ 157us) server_negotiation.cc:299] Negotiation successful
0925 22:36:18.262887 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":82,"threads_started":1}
Sep 25 22:36:18 dist-test-slave-wjdr krb5kdc[10678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375778, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
I20260925 22:36:18.273720 10762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:18.265893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43370 (local address 127.6.246.254:45291)
0925 22:36:18.266029 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:18.266035 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:18.266099 (+ 64us) server_negotiation.cc:407] Connection header received
0925 22:36:18.266241 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:18.266244 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:18.266292 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:18.266384 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:18.266945 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:18.267452 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:18.268246 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:18.268547 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:18.268671 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:18.270880 (+ 2209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:18.270883 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:18.270886 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:18.270910 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:18.272108 (+ 1198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:18.272520 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:18.272525 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:18.272527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:18.272608 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:18.272870 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:18.272874 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:18.272877 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:18.273086 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:18.273222 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:18.273438 (+ 216us) server_negotiation.cc:299] Negotiation successful
0925 22:36:18.273551 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260925 22:36:18.274123 10723 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43370
I20260925 22:36:18.274708 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 10785
I20260925 22:36:18.281162 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 10919
I20260925 22:36:18.287523 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 11053
I20260925 22:36:18.292874 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 10691
2026-09-25T22:36:18Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2108 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:36:18 dist-test-slave-wjdr krb5kdc[11194](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:36:18 dist-test-slave-wjdr krb5kdc[11194](info): set up 2 sockets
Sep 25 22:36:18 dist-test-slave-wjdr krb5kdc[11194](info): commencing operation
krb5kdc: starting...
W20260925 22:36:19.332828 7131 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Sep 25 22:36:19 dist-test-slave-wjdr krb5kdc[11194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:36:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:36:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.246.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab.
Entry for principal kudu/127.6.246.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab.
I20260925 22:36:19.451594 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:45707
--webserver_interface=127.6.246.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab
--principal=kudu/127.6.246.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:42977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.246.254:45707
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:36:19.558591 11207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:36:19.558873 11207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:36:19.558954 11207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:36:19.562978 11207 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:36:19.563069 11207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:36:19.563107 11207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:36:19.563144 11207 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:36:19.563175 11207 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:36:19.568271 11207 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:42977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.246.254:45707
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab
--principal=kudu/127.6.246.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:45707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.246.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11207
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:36:19.569346 11207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:36:19.570362 11207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:36:19.578358 11212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:19.578397 11215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:19.578642 11207 server_base.cc:1034] running on GCE node
W20260925 22:36:19.578739 11213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:19.579380 11207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:36:19.580660 11207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:36:19.581846 11207 hybrid_clock.cc:648] HybridClock initialized: now 1790375779581826 us; error 33 us; skew 500 ppm
Sep 25 22:36:19 dist-test-slave-wjdr krb5kdc[11194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375779, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:36:19.584589 11207 init.cc:377] Logged in from keytab as kudu/127.6.246.254@KRBTEST.COM (short username kudu)
I20260925 22:36:19.585323 11207 fs_manager.cc:359] Metadata directory not provided
I20260925 22:36:19.585424 11207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:36:19.585645 11207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:36:19.587697 11207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/minicluster-data/master-0/data/instance:
uuid: "49ac656251a84d86be24cf9ea223666c"
format_stamp: "Formatted at 2026-09-25 22:36:19 on dist-test-slave-wjdr"
server_key: "c4446b98ad8e44ed88616002821a80b2"
server_key_iv: "02ee9b104d44adc0a2c9c7556fafa5a8"
server_key_version: "encryptionkey@0"
I20260925 22:36:19.588249 11207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/minicluster-data/master-0/wal/instance:
uuid: "49ac656251a84d86be24cf9ea223666c"
format_stamp: "Formatted at 2026-09-25 22:36:19 on dist-test-slave-wjdr"
server_key: "c4446b98ad8e44ed88616002821a80b2"
server_key_iv: "02ee9b104d44adc0a2c9c7556fafa5a8"
server_key_version: "encryptionkey@0"
I20260925 22:36:19.591732 11207 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:36:19.594022 11221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:19.595111 11207 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 22:36:19.595248 11207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/minicluster-data/master-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/minicluster-data/master-0/wal
uuid: "49ac656251a84d86be24cf9ea223666c"
format_stamp: "Formatted at 2026-09-25 22:36:19 on dist-test-slave-wjdr"
server_key: "c4446b98ad8e44ed88616002821a80b2"
server_key_iv: "02ee9b104d44adc0a2c9c7556fafa5a8"
server_key_version: "encryptionkey@0"
I20260925 22:36:19.595355 11207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:36:19.609236 11207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:36:19.611884 11207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:36:19.612067 11207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:36:19.618822 11207 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.254:45707
I20260925 22:36:19.619148 11273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.254:45707 every 8 connection(s)
I20260925 22:36:19.619481 11207 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
I20260925 22:36:19.622123 11274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:36:19.627354 11274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49ac656251a84d86be24cf9ea223666c: Bootstrap starting.
I20260925 22:36:19.627421 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 11207
I20260925 22:36:19.627539 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/minicluster-data/master-0/wal/instance
I20260925 22:36:19.627810 7131 external_mini_cluster.cc:1428] Setting key ee6e41b287a46ec7a24b4a28a830aa98
I20260925 22:36:19.629842 11274 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49ac656251a84d86be24cf9ea223666c: Neither blocks nor log segments found. Creating new log.
I20260925 22:36:19.630579 11274 log.cc:824] T 00000000000000000000000000000000 P 49ac656251a84d86be24cf9ea223666c: Log is configured to *not* fsync() on all Append() calls
I20260925 22:36:19.632438 11274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49ac656251a84d86be24cf9ea223666c: No bootstrap required, opened a new log
Sep 25 22:36:19 dist-test-slave-wjdr krb5kdc[11194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
I20260925 22:36:19.635922 11274 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49ac656251a84d86be24cf9ea223666c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49ac656251a84d86be24cf9ea223666c" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 45707 } }
I20260925 22:36:19.636202 11274 raft_consensus.cc:374] T 00000000000000000000000000000000 P 49ac656251a84d86be24cf9ea223666c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:36:19.636291 11274 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49ac656251a84d86be24cf9ea223666c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49ac656251a84d86be24cf9ea223666c, State: Initialized, Role: FOLLOWER
I20260925 22:36:19.636801 11274 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49ac656251a84d86be24cf9ea223666c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49ac656251a84d86be24cf9ea223666c" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 45707 } }
I20260925 22:36:19.636979 11274 raft_consensus.cc:388] T 00000000000000000000000000000000 P 49ac656251a84d86be24cf9ea223666c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:36:19.637046 11274 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49ac656251a84d86be24cf9ea223666c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:36:19.637144 11274 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49ac656251a84d86be24cf9ea223666c [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:36:19.638309 11274 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49ac656251a84d86be24cf9ea223666c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49ac656251a84d86be24cf9ea223666c" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 45707 } }
I20260925 22:36:19.638667 11274 leader_election.cc:302] T 00000000000000000000000000000000 P 49ac656251a84d86be24cf9ea223666c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 49ac656251a84d86be24cf9ea223666c; no voters:
I20260925 22:36:19.639006 11274 leader_election.cc:288] T 00000000000000000000000000000000 P 49ac656251a84d86be24cf9ea223666c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:36:19.639065 11279 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49ac656251a84d86be24cf9ea223666c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:36:19.639278 11279 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49ac656251a84d86be24cf9ea223666c [term 1 LEADER]: Becoming Leader. State: Replica: 49ac656251a84d86be24cf9ea223666c, State: Running, Role: LEADER
I20260925 22:36:19.639715 11279 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49ac656251a84d86be24cf9ea223666c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49ac656251a84d86be24cf9ea223666c" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 45707 } }
I20260925 22:36:19.640177 11274 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49ac656251a84d86be24cf9ea223666c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:36:19.640985 11280 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49ac656251a84d86be24cf9ea223666c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49ac656251a84d86be24cf9ea223666c. Latest consensus state: current_term: 1 leader_uuid: "49ac656251a84d86be24cf9ea223666c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49ac656251a84d86be24cf9ea223666c" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 45707 } } }
I20260925 22:36:19.641095 11280 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49ac656251a84d86be24cf9ea223666c [sys.catalog]: This master's current role is: LEADER
I20260925 22:36:19.641366 11281 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49ac656251a84d86be24cf9ea223666c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "49ac656251a84d86be24cf9ea223666c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49ac656251a84d86be24cf9ea223666c" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 45707 } } }
I20260925 22:36:19.641469 11281 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49ac656251a84d86be24cf9ea223666c [sys.catalog]: This master's current role is: LEADER
I20260925 22:36:19.641916 11277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:19.629027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41014 (local address 127.6.246.254:45707)
0925 22:36:19.629409 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:19.629424 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:19.629456 (+ 32us) server_negotiation.cc:407] Connection header received
0925 22:36:19.630098 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:19.630114 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:19.630443 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:19.630730 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:19.631551 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:19.632557 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:19.633441 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:19.633934 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:19.634099 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:19.636605 (+ 2506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:19.636611 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:19.636626 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:19.636655 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:19.638700 (+ 2045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:19.639183 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:19.639188 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:19.639192 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:19.639346 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:19.639648 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:19.639654 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:19.639658 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:19.640140 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:19.640343 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:19.640681 (+ 338us) server_negotiation.cc:299] Negotiation successful
0925 22:36:19.640992 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":124,"threads_started":1}
W20260925 22:36:19.645301 11294 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 49ac656251a84d86be24cf9ea223666c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:36:19.645383 11294 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:36:19.645469 11288 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:36:19.646149 11288 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:36:19.651325 11288 catalog_manager.cc:1347] Generated new cluster ID: ed8d7bf1ba2c4047a3781a59f7896344
I20260925 22:36:19.651394 11288 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:36:19.662788 11288 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:36:19.663616 11288 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:36:19.673609 11288 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 49ac656251a84d86be24cf9ea223666c: Generated new TSK 0
I20260925 22:36:19.674405 11288 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.246.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.193.keytab.
Entry for principal kudu/127.6.246.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.193.keytab.
I20260925 22:36:19.705588 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.193:0
--local_ip_for_outbound_sockets=127.6.246.193
--webserver_interface=127.6.246.193
--webserver_port=0
--tserver_master_addrs=127.6.246.254:45707
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.193.keytab
--principal=kudu/127.6.246.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:42977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:36:19.815956 11300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:36:19.816197 11300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:36:19.816272 11300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:36:19.820237 11300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:36:19.820338 11300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:36:19.820446 11300 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.193
I20260925 22:36:19.825573 11300 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:42977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.193.keytab
--principal=kudu/127.6.246.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.246.193
--webserver_port=0
--tserver_master_addrs=127.6.246.254:45707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11300
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.193
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:36:19.826552 11300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:36:19.827513 11300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:36:19.836675 11306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:19.836707 11305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:19.836772 11300 server_base.cc:1034] running on GCE node
W20260925 22:36:19.836707 11308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:19.837296 11300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:36:19.837855 11300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:36:19.839010 11300 hybrid_clock.cc:648] HybridClock initialized: now 1790375779838999 us; error 29 us; skew 500 ppm
Sep 25 22:36:19 dist-test-slave-wjdr krb5kdc[11194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375779, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:36:19.842032 11300 init.cc:377] Logged in from keytab as kudu/127.6.246.193@KRBTEST.COM (short username kudu)
I20260925 22:36:19.842763 11300 fs_manager.cc:359] Metadata directory not provided
I20260925 22:36:19.842855 11300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:36:19.843081 11300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:36:19.845077 11300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/minicluster-data/ts-0/data/instance:
uuid: "ef922b98e358410186dadab999b93698"
format_stamp: "Formatted at 2026-09-25 22:36:19 on dist-test-slave-wjdr"
server_key: "86c7f3647996744d9edaf111177b4ffd"
server_key_iv: "76cb7b6f0d6fb468d106ec788be1693e"
server_key_version: "encryptionkey@0"
I20260925 22:36:19.845615 11300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/minicluster-data/ts-0/wal/instance:
uuid: "ef922b98e358410186dadab999b93698"
format_stamp: "Formatted at 2026-09-25 22:36:19 on dist-test-slave-wjdr"
server_key: "86c7f3647996744d9edaf111177b4ffd"
server_key_iv: "76cb7b6f0d6fb468d106ec788be1693e"
server_key_version: "encryptionkey@0"
I20260925 22:36:19.849028 11300 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:36:19.851261 11314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:19.852340 11300 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:36:19.852501 11300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/minicluster-data/ts-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/minicluster-data/ts-0/wal
uuid: "ef922b98e358410186dadab999b93698"
format_stamp: "Formatted at 2026-09-25 22:36:19 on dist-test-slave-wjdr"
server_key: "86c7f3647996744d9edaf111177b4ffd"
server_key_iv: "76cb7b6f0d6fb468d106ec788be1693e"
server_key_version: "encryptionkey@0"
I20260925 22:36:19.852612 11300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:36:19.869338 11300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:36:19.871943 11300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:36:19.872123 11300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:36:19.872651 11300 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:36:19.873400 11300 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:36:19.873466 11300 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:19.873528 11300 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:36:19.873576 11300 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:19.882085 11300 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.193:39767
I20260925 22:36:19.882102 11427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.193:39767 every 8 connection(s)
I20260925 22:36:19.882617 11300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
Sep 25 22:36:19 dist-test-slave-wjdr krb5kdc[11194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375779, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.193@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
I20260925 22:36:19.891489 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 11300
I20260925 22:36:19.891623 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/minicluster-data/ts-0/wal/instance
I20260925 22:36:19.891916 7131 external_mini_cluster.cc:1428] Setting key acedd94e53bc5e67b4f0db3b3d5165d7
I20260925 22:36:19.895360 11277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:19.884200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.193:54145 (local address 127.6.246.254:45707)
0925 22:36:19.884357 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:19.884363 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:19.885230 (+ 867us) server_negotiation.cc:407] Connection header received
0925 22:36:19.886425 (+ 1195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:19.886428 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:19.886473 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:19.886545 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:19.887599 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:19.888105 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:19.889260 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:19.889483 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:19.889560 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:19.891857 (+ 2297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:19.891860 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:19.891862 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:19.891883 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:19.893473 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:19.893964 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:19.893968 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:19.893970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:19.894052 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:19.894377 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:19.894382 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:19.894384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:19.894554 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:19.894655 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:19.895090 (+ 435us) server_negotiation.cc:299] Negotiation successful
0925 22:36:19.895200 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 22:36:19.896061 11430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:19.884563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:45707 (local address 127.6.246.193:54145)
0925 22:36:19.885088 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:19.885129 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:19.886240 (+ 1111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:19.886690 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:19.886696 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:19.887044 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:19.887469 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:19.887480 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:19.888206 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:19.888208 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:19.889162 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:19.889172 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:19.889600 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:19.889603 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:19.889732 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:19.890221 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:19.890237 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:19.891676 (+ 1439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:36:19.893583 (+ 1907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:19.893589 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:19.893601 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:19.893861 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:19.894146 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:19.894149 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:19.894151 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:19.894287 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:19.894653 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:36:19.894659 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:36:19.894878 (+ 219us) client_negotiation.cc:769] Sending connection context
0925 22:36:19.895049 (+ 171us) client_negotiation.cc:240] Negotiation successful
0925 22:36:19.895340 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":100,"threads_started":1}
I20260925 22:36:19.897223 11428 heartbeater.cc:347] Connected to a master server at 127.6.246.254:45707
I20260925 22:36:19.897468 11428 heartbeater.cc:464] Registering TS with master...
I20260925 22:36:19.897993 11428 heartbeater.cc:511] Master 127.6.246.254:45707 requested a full tablet report, sending...
I20260925 22:36:19.899766 11238 ts_manager.cc:194] Registered new tserver with Master: ef922b98e358410186dadab999b93698 (127.6.246.193:39767)
I20260925 22:36:19.901419 11238 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.246.193@KRBTEST.COM'} at 127.6.246.193:54145
WARNING: no policy specified for kudu/127.6.246.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.194.keytab.
Entry for principal kudu/127.6.246.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.194.keytab.
I20260925 22:36:19.917532 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.194:0
--local_ip_for_outbound_sockets=127.6.246.194
--webserver_interface=127.6.246.194
--webserver_port=0
--tserver_master_addrs=127.6.246.254:45707
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.194.keytab
--principal=kudu/127.6.246.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:42977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:36:20.027486 11433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:36:20.027767 11433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:36:20.027859 11433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:36:20.031956 11433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:36:20.032055 11433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:36:20.032150 11433 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.194
I20260925 22:36:20.037352 11433 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:42977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.194.keytab
--principal=kudu/127.6.246.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.246.194
--webserver_port=0
--tserver_master_addrs=127.6.246.254:45707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.194
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:36:20.038404 11433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:36:20.039410 11433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:36:20.051429 11439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:20.052747 11441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:20.052958 11438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:20.053194 11433 server_base.cc:1034] running on GCE node
I20260925 22:36:20.053810 11433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:36:20.054574 11433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:36:20.055711 11433 hybrid_clock.cc:648] HybridClock initialized: now 1790375780055683 us; error 41 us; skew 500 ppm
Sep 25 22:36:20 dist-test-slave-wjdr krb5kdc[11194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375780, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:36:20.058881 11433 init.cc:377] Logged in from keytab as kudu/127.6.246.194@KRBTEST.COM (short username kudu)
I20260925 22:36:20.059916 11433 fs_manager.cc:359] Metadata directory not provided
I20260925 22:36:20.060016 11433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:36:20.060257 11433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:36:20.063005 11433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/minicluster-data/ts-1/data/instance:
uuid: "7b751a5466824e6da229fbeb41963618"
format_stamp: "Formatted at 2026-09-25 22:36:20 on dist-test-slave-wjdr"
server_key: "b29eec77eb8061ab71f0a833f1028cf4"
server_key_iv: "870b65b3946f2f4bf26477057dcd9738"
server_key_version: "encryptionkey@0"
I20260925 22:36:20.063594 11433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/minicluster-data/ts-1/wal/instance:
uuid: "7b751a5466824e6da229fbeb41963618"
format_stamp: "Formatted at 2026-09-25 22:36:20 on dist-test-slave-wjdr"
server_key: "b29eec77eb8061ab71f0a833f1028cf4"
server_key_iv: "870b65b3946f2f4bf26477057dcd9738"
server_key_version: "encryptionkey@0"
I20260925 22:36:20.068585 11433 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260925 22:36:20.071943 11447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:20.073500 11433 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:36:20.073625 11433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/minicluster-data/ts-1/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/minicluster-data/ts-1/wal
uuid: "7b751a5466824e6da229fbeb41963618"
format_stamp: "Formatted at 2026-09-25 22:36:20 on dist-test-slave-wjdr"
server_key: "b29eec77eb8061ab71f0a833f1028cf4"
server_key_iv: "870b65b3946f2f4bf26477057dcd9738"
server_key_version: "encryptionkey@0"
I20260925 22:36:20.073757 11433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:36:20.090205 11433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:36:20.093325 11433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:36:20.093544 11433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:36:20.094162 11433 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:36:20.094967 11433 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:36:20.095017 11433 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:20.095057 11433 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:36:20.095081 11433 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:20.104130 11433 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.194:32929
I20260925 22:36:20.104143 11560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.194:32929 every 8 connection(s)
I20260925 22:36:20.104811 11433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
I20260925 22:36:20.113565 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 11433
I20260925 22:36:20.113708 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/minicluster-data/ts-1/wal/instance
I20260925 22:36:20.114087 7131 external_mini_cluster.cc:1428] Setting key 98b4c65dc1aa4b815bda8219db28a6de
Sep 25 22:36:20 dist-test-slave-wjdr krb5kdc[11194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375780, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.194@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
I20260925 22:36:20.118639 11277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:20.106518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.194:49633 (local address 127.6.246.254:45707)
0925 22:36:20.106658 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:20.106661 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:20.107494 (+ 833us) server_negotiation.cc:407] Connection header received
0925 22:36:20.108829 (+ 1335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:20.108833 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:20.108909 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:20.109003 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:20.110120 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:20.110645 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:20.111763 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:20.111991 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:20.112098 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:20.114772 (+ 2674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:20.114775 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:20.114778 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:20.114806 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:20.116293 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:20.116868 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:20.116874 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:20.116878 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:20.116984 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:20.117477 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:20.117480 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:20.117481 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:20.117658 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:20.117748 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:20.118412 (+ 664us) server_negotiation.cc:299] Negotiation successful
0925 22:36:20.118523 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 22:36:20.119771 11563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:20.106921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:45707 (local address 127.6.246.194:49633)
0925 22:36:20.107349 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:20.107396 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:20.108627 (+ 1231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:20.109167 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:20.109174 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:20.109524 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:20.109981 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:20.109994 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:20.110770 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:20.110779 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:20.111639 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:20.111647 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:20.112141 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:20.112145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:20.112308 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:20.112950 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:20.112968 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:20.114605 (+ 1637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:36:20.116430 (+ 1825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:20.116437 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:20.116468 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:20.116737 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:20.117086 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:20.117091 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:20.117094 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:20.117278 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:20.117779 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:36:20.117787 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:36:20.118117 (+ 330us) client_negotiation.cc:769] Sending connection context
0925 22:36:20.118316 (+ 199us) client_negotiation.cc:240] Negotiation successful
0925 22:36:20.118661 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":114,"threads_started":1}
I20260925 22:36:20.121083 11561 heartbeater.cc:347] Connected to a master server at 127.6.246.254:45707
I20260925 22:36:20.121364 11561 heartbeater.cc:464] Registering TS with master...
I20260925 22:36:20.121850 11561 heartbeater.cc:511] Master 127.6.246.254:45707 requested a full tablet report, sending...
I20260925 22:36:20.122992 11238 ts_manager.cc:194] Registered new tserver with Master: 7b751a5466824e6da229fbeb41963618 (127.6.246.194:32929)
WARNING: no policy specified for kudu/127.6.246.195@KRBTEST.COM; defaulting to no policy
I20260925 22:36:20.123582 11238 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.246.194@KRBTEST.COM'} at 127.6.246.194:49633
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.195.keytab.
Entry for principal kudu/127.6.246.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.195.keytab.
I20260925 22:36:20.140950 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.195:0
--local_ip_for_outbound_sockets=127.6.246.195
--webserver_interface=127.6.246.195
--webserver_port=0
--tserver_master_addrs=127.6.246.254:45707
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.195.keytab
--principal=kudu/127.6.246.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:42977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:36:20.249605 11566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:36:20.249845 11566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:36:20.249889 11566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:36:20.253969 11566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:36:20.254024 11566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:36:20.254122 11566 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.195
I20260925 22:36:20.259346 11566 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:42977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.195.keytab
--principal=kudu/127.6.246.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.246.195
--webserver_port=0
--tserver_master_addrs=127.6.246.254:45707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11566
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.195
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:36:20.260349 11566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:36:20.261338 11566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:36:20.270288 11571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:20.270234 11572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:20.270265 11574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:20.270593 11566 server_base.cc:1034] running on GCE node
I20260925 22:36:20.271023 11566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:36:20.271682 11566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:36:20.272864 11566 hybrid_clock.cc:648] HybridClock initialized: now 1790375780272845 us; error 31 us; skew 500 ppm
Sep 25 22:36:20 dist-test-slave-wjdr krb5kdc[11194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375780, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:36:20.275645 11566 init.cc:377] Logged in from keytab as kudu/127.6.246.195@KRBTEST.COM (short username kudu)
I20260925 22:36:20.276501 11566 fs_manager.cc:359] Metadata directory not provided
I20260925 22:36:20.276597 11566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:36:20.276799 11566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:36:20.278836 11566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/minicluster-data/ts-2/data/instance:
uuid: "47690fe7d3d841c0b59693877c337108"
format_stamp: "Formatted at 2026-09-25 22:36:20 on dist-test-slave-wjdr"
server_key: "b05a96522c3b05ed612e0772803a9b6f"
server_key_iv: "1549d8b9245c089d5947a7a3416a7b2e"
server_key_version: "encryptionkey@0"
I20260925 22:36:20.279413 11566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/minicluster-data/ts-2/wal/instance:
uuid: "47690fe7d3d841c0b59693877c337108"
format_stamp: "Formatted at 2026-09-25 22:36:20 on dist-test-slave-wjdr"
server_key: "b05a96522c3b05ed612e0772803a9b6f"
server_key_iv: "1549d8b9245c089d5947a7a3416a7b2e"
server_key_version: "encryptionkey@0"
I20260925 22:36:20.283245 11566 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260925 22:36:20.285661 11580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:20.286968 11566 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 22:36:20.287103 11566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/minicluster-data/ts-2/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/minicluster-data/ts-2/wal
uuid: "47690fe7d3d841c0b59693877c337108"
format_stamp: "Formatted at 2026-09-25 22:36:20 on dist-test-slave-wjdr"
server_key: "b05a96522c3b05ed612e0772803a9b6f"
server_key_iv: "1549d8b9245c089d5947a7a3416a7b2e"
server_key_version: "encryptionkey@0"
I20260925 22:36:20.287211 11566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:36:20.311646 11566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:36:20.314924 11566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:36:20.315119 11566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:36:20.315691 11566 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:36:20.316605 11566 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:36:20.316671 11566 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:20.316731 11566 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:36:20.316789 11566 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:20.325332 11566 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.195:42693
I20260925 22:36:20.325348 11693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.195:42693 every 8 connection(s)
I20260925 22:36:20.325915 11566 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
I20260925 22:36:20.327118 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 11566
I20260925 22:36:20.327258 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/minicluster-data/ts-2/wal/instance
I20260925 22:36:20.327598 7131 external_mini_cluster.cc:1428] Setting key 9a70bc7806112fc74b042d58aa10b145
Sep 25 22:36:20 dist-test-slave-wjdr krb5kdc[11194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375780, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.195@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
I20260925 22:36:20.339133 11277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:20.327710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.195:51155 (local address 127.6.246.254:45707)
0925 22:36:20.327852 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:20.327858 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:20.328717 (+ 859us) server_negotiation.cc:407] Connection header received
0925 22:36:20.330070 (+ 1353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:20.330073 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:20.330120 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:20.330193 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:20.331284 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:20.331820 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:20.332991 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:20.333232 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:20.333332 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:20.335718 (+ 2386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:20.335727 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:20.335729 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:20.335751 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:20.337176 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:20.337668 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:20.337673 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:20.337675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:20.337755 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:20.338093 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:20.338097 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:20.338098 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:20.338276 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:20.338382 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:20.338846 (+ 464us) server_negotiation.cc:299] Negotiation successful
0925 22:36:20.338959 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 22:36:20.339886 11696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:20.328085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:45707 (local address 127.6.246.195:51155)
0925 22:36:20.328560 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:20.328610 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:20.329853 (+ 1243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:20.330353 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:20.330359 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:20.330688 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:20.331155 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:20.331168 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:20.331921 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:20.331923 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:20.332861 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:20.332870 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:20.333364 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:20.333367 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:20.333492 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:20.334080 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:20.334095 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:20.335550 (+ 1455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:36:20.337299 (+ 1749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:20.337305 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:20.337318 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:20.337565 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:20.337854 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:20.337859 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:20.337861 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:20.337992 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:20.338379 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:36:20.338386 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:36:20.338619 (+ 233us) client_negotiation.cc:769] Sending connection context
0925 22:36:20.338801 (+ 182us) client_negotiation.cc:240] Negotiation successful
0925 22:36:20.339096 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":136,"threads_started":1}
I20260925 22:36:20.341053 11694 heartbeater.cc:347] Connected to a master server at 127.6.246.254:45707
I20260925 22:36:20.341452 11694 heartbeater.cc:464] Registering TS with master...
I20260925 22:36:20.341913 11694 heartbeater.cc:511] Master 127.6.246.254:45707 requested a full tablet report, sending...
I20260925 22:36:20.342819 11237 ts_manager.cc:194] Registered new tserver with Master: 47690fe7d3d841c0b59693877c337108 (127.6.246.195:42693)
I20260925 22:36:20.343365 11237 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.246.195@KRBTEST.COM'} at 127.6.246.195:51155
I20260925 22:36:20.353380 7131 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:36:20.363101 11277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:20.355287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41024 (local address 127.6.246.254:45707)
0925 22:36:20.355470 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:20.355476 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:20.355739 (+ 263us) server_negotiation.cc:407] Connection header received
0925 22:36:20.355884 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:20.355887 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:20.355937 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:20.356019 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:20.356771 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:20.357302 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:20.358138 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:20.358362 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:20.358475 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:20.359845 (+ 1370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:20.359849 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:20.359851 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:20.359892 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:20.361405 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:20.361831 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:20.361837 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:20.361840 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:20.361925 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:20.362198 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:20.362202 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:20.362204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:20.362428 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:20.362586 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:20.362822 (+ 236us) server_negotiation.cc:299] Negotiation successful
0925 22:36:20.362932 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260925 22:36:20.366056 11237 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41024:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:36:20.369064 11237 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:36:20.383252 11710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:20.378241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38450 (local address 127.6.246.195:42693)
0925 22:36:20.378588 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:20.378605 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:20.378662 (+ 57us) server_negotiation.cc:407] Connection header received
0925 22:36:20.378738 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:20.378743 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:20.378919 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:20.379072 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:36:20.379337 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:20.380485 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:20.381758 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:20.382549 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:20.382676 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:20.382837 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:20.382984 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 22:36:20.383086 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":184,"threads_started":1}
I20260925 22:36:20.383247 11705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:20.377171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.194:32929 (local address 127.0.0.1:37378)
0925 22:36:20.377650 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:20.377696 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:20.377826 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:20.378913 (+ 1087us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:20.378919 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:20.378941 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:36:20.379077 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:20.379087 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:20.380351 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:20.380355 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:20.381747 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:20.381758 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:20.382742 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:20.382746 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:20.382843 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:20.382870 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 22:36:20.382921 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 22:36:20.383101 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"mutex_wait_us":47,"thread_start_us":65,"threads_started":1}
I20260925 22:36:20.383266 11708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:20.377343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.195:42693 (local address 127.0.0.1:38450)
0925 22:36:20.378099 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:20.378112 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:20.378437 (+ 325us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:20.379052 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:20.379055 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:20.379073 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:36:20.379168 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:20.379176 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:20.380625 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:20.380629 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:20.381641 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:20.381650 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:20.382670 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:20.382674 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:20.382782 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:20.382830 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 22:36:20.382947 (+ 117us) client_negotiation.cc:240] Negotiation successful
0925 22:36:20.383101 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":698,"mutex_wait_us":42,"thread_start_us":86,"threads_started":1}
I20260925 22:36:20.384099 11707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:20.377305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37378 (local address 127.6.246.194:32929)
0925 22:36:20.378513 (+ 1208us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:20.378526 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:20.378548 (+ 22us) server_negotiation.cc:407] Connection header received
0925 22:36:20.378616 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:20.378619 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:20.378766 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:20.378925 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:36:20.379216 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:20.380228 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:20.381870 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:20.382615 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:20.383692 (+ 1077us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:20.383828 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:20.383873 (+ 45us) server_negotiation.cc:299] Negotiation successful
0925 22:36:20.383954 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1100,"thread_start_us":123,"threads_started":1}
I20260925 22:36:20.384435 11709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:20.378058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38864 (local address 127.6.246.193:39767)
0925 22:36:20.378330 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:20.378337 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:20.379253 (+ 916us) server_negotiation.cc:407] Connection header received
0925 22:36:20.379408 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:20.379413 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:20.379577 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:20.379725 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:36:20.379959 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:20.381110 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:20.382279 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:20.383156 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:20.383254 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:20.383406 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:20.383909 (+ 503us) server_negotiation.cc:299] Negotiation successful
0925 22:36:20.384008 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":93,"threads_started":1}
I20260925 22:36:20.384718 11706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:20.377861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.193:39767 (local address 127.0.0.1:38864)
0925 22:36:20.379162 (+ 1301us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:20.379176 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:20.379267 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:20.379733 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:20.379736 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:20.379754 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:36:20.379851 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:20.379857 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:20.381242 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:20.381246 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:20.382167 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:20.382174 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:20.383646 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:20.383650 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:20.383741 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:20.383763 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 22:36:20.384503 (+ 740us) client_negotiation.cc:240] Negotiation successful
0925 22:36:20.384594 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1227,"thread_start_us":83,"threads_started":1}
I20260925 22:36:20.386062 11628 tablet_service.cc:1467] Processing CreateTablet for tablet 5e6a8fbaa28c40a89e0828e7af7fba25 (DEFAULT_TABLE table=test-table [id=f28e1ca587f943feafa8ea355433ea5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:36:20.386991 11628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e6a8fbaa28c40a89e0828e7af7fba25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:36:20.386773 11362 tablet_service.cc:1467] Processing CreateTablet for tablet 5e6a8fbaa28c40a89e0828e7af7fba25 (DEFAULT_TABLE table=test-table [id=f28e1ca587f943feafa8ea355433ea5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:36:20.387629 11362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e6a8fbaa28c40a89e0828e7af7fba25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:36:20.390964 11495 tablet_service.cc:1467] Processing CreateTablet for tablet 5e6a8fbaa28c40a89e0828e7af7fba25 (DEFAULT_TABLE table=test-table [id=f28e1ca587f943feafa8ea355433ea5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:36:20.391925 11495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e6a8fbaa28c40a89e0828e7af7fba25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:36:20.393893 11711 tablet_bootstrap.cc:492] T 5e6a8fbaa28c40a89e0828e7af7fba25 P ef922b98e358410186dadab999b93698: Bootstrap starting.
I20260925 22:36:20.395224 11712 tablet_bootstrap.cc:492] T 5e6a8fbaa28c40a89e0828e7af7fba25 P 47690fe7d3d841c0b59693877c337108: Bootstrap starting.
I20260925 22:36:20.396207 11711 tablet_bootstrap.cc:654] T 5e6a8fbaa28c40a89e0828e7af7fba25 P ef922b98e358410186dadab999b93698: Neither blocks nor log segments found. Creating new log.
I20260925 22:36:20.397505 11712 tablet_bootstrap.cc:654] T 5e6a8fbaa28c40a89e0828e7af7fba25 P 47690fe7d3d841c0b59693877c337108: Neither blocks nor log segments found. Creating new log.
I20260925 22:36:20.397581 11711 log.cc:824] T 5e6a8fbaa28c40a89e0828e7af7fba25 P ef922b98e358410186dadab999b93698: Log is configured to *not* fsync() on all Append() calls
I20260925 22:36:20.397948 11713 tablet_bootstrap.cc:492] T 5e6a8fbaa28c40a89e0828e7af7fba25 P 7b751a5466824e6da229fbeb41963618: Bootstrap starting.
I20260925 22:36:20.398756 11712 log.cc:824] T 5e6a8fbaa28c40a89e0828e7af7fba25 P 47690fe7d3d841c0b59693877c337108: Log is configured to *not* fsync() on all Append() calls
I20260925 22:36:20.399967 11713 tablet_bootstrap.cc:654] T 5e6a8fbaa28c40a89e0828e7af7fba25 P 7b751a5466824e6da229fbeb41963618: Neither blocks nor log segments found. Creating new log.
I20260925 22:36:20.400658 11712 tablet_bootstrap.cc:492] T 5e6a8fbaa28c40a89e0828e7af7fba25 P 47690fe7d3d841c0b59693877c337108: No bootstrap required, opened a new log
I20260925 22:36:20.400784 11711 tablet_bootstrap.cc:492] T 5e6a8fbaa28c40a89e0828e7af7fba25 P ef922b98e358410186dadab999b93698: No bootstrap required, opened a new log
I20260925 22:36:20.400882 11712 ts_tablet_manager.cc:1397] T 5e6a8fbaa28c40a89e0828e7af7fba25 P 47690fe7d3d841c0b59693877c337108: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 22:36:20.400956 11711 ts_tablet_manager.cc:1397] T 5e6a8fbaa28c40a89e0828e7af7fba25 P ef922b98e358410186dadab999b93698: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.001s
I20260925 22:36:20.401120 11713 log.cc:824] T 5e6a8fbaa28c40a89e0828e7af7fba25 P 7b751a5466824e6da229fbeb41963618: Log is configured to *not* fsync() on all Append() calls
I20260925 22:36:20.402901 11713 tablet_bootstrap.cc:492] T 5e6a8fbaa28c40a89e0828e7af7fba25 P 7b751a5466824e6da229fbeb41963618: No bootstrap required, opened a new log
I20260925 22:36:20.403059 11713 ts_tablet_manager.cc:1397] T 5e6a8fbaa28c40a89e0828e7af7fba25 P 7b751a5466824e6da229fbeb41963618: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 22:36:20.403940 11712 raft_consensus.cc:348] T 5e6a8fbaa28c40a89e0828e7af7fba25 P 47690fe7d3d841c0b59693877c337108 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b751a5466824e6da229fbeb41963618" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 32929 } } peers { permanent_uuid: "47690fe7d3d841c0b59693877c337108" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 42693 } } peers { permanent_uuid: "ef922b98e358410186dadab999b93698" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 39767 } }
I20260925 22:36:20.404177 11712 raft_consensus.cc:374] T 5e6a8fbaa28c40a89e0828e7af7fba25 P 47690fe7d3d841c0b59693877c337108 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:36:20.404230 11712 raft_consensus.cc:729] T 5e6a8fbaa28c40a89e0828e7af7fba25 P 47690fe7d3d841c0b59693877c337108 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47690fe7d3d841c0b59693877c337108, State: Initialized, Role: FOLLOWER
I20260925 22:36:20.404487 11711 raft_consensus.cc:348] T 5e6a8fbaa28c40a89e0828e7af7fba25 P ef922b98e358410186dadab999b93698 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b751a5466824e6da229fbeb41963618" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 32929 } } peers { permanent_uuid: "47690fe7d3d841c0b59693877c337108" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 42693 } } peers { permanent_uuid: "ef922b98e358410186dadab999b93698" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 39767 } }
I20260925 22:36:20.404722 11711 raft_consensus.cc:374] T 5e6a8fbaa28c40a89e0828e7af7fba25 P ef922b98e358410186dadab999b93698 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:36:20.404790 11711 raft_consensus.cc:729] T 5e6a8fbaa28c40a89e0828e7af7fba25 P ef922b98e358410186dadab999b93698 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef922b98e358410186dadab999b93698, State: Initialized, Role: FOLLOWER
I20260925 22:36:20.404671 11712 consensus_queue.cc:260] T 5e6a8fbaa28c40a89e0828e7af7fba25 P 47690fe7d3d841c0b59693877c337108 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b751a5466824e6da229fbeb41963618" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 32929 } } peers { permanent_uuid: "47690fe7d3d841c0b59693877c337108" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 42693 } } peers { permanent_uuid: "ef922b98e358410186dadab999b93698" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 39767 } }
I20260925 22:36:20.405174 11711 consensus_queue.cc:260] T 5e6a8fbaa28c40a89e0828e7af7fba25 P ef922b98e358410186dadab999b93698 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b751a5466824e6da229fbeb41963618" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 32929 } } peers { permanent_uuid: "47690fe7d3d841c0b59693877c337108" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 42693 } } peers { permanent_uuid: "ef922b98e358410186dadab999b93698" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 39767 } }
I20260925 22:36:20.405918 11712 ts_tablet_manager.cc:1428] T 5e6a8fbaa28c40a89e0828e7af7fba25 P 47690fe7d3d841c0b59693877c337108: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260925 22:36:20.406040 11711 ts_tablet_manager.cc:1428] T 5e6a8fbaa28c40a89e0828e7af7fba25 P ef922b98e358410186dadab999b93698: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
I20260925 22:36:20.406325 11694 heartbeater.cc:503] Master 127.6.246.254:45707 was elected leader, sending a full tablet report...
I20260925 22:36:20.406381 11428 heartbeater.cc:503] Master 127.6.246.254:45707 was elected leader, sending a full tablet report...
I20260925 22:36:20.406649 11713 raft_consensus.cc:348] T 5e6a8fbaa28c40a89e0828e7af7fba25 P 7b751a5466824e6da229fbeb41963618 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b751a5466824e6da229fbeb41963618" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 32929 } } peers { permanent_uuid: "47690fe7d3d841c0b59693877c337108" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 42693 } } peers { permanent_uuid: "ef922b98e358410186dadab999b93698" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 39767 } }
I20260925 22:36:20.406874 11713 raft_consensus.cc:374] T 5e6a8fbaa28c40a89e0828e7af7fba25 P 7b751a5466824e6da229fbeb41963618 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:36:20.406919 11713 raft_consensus.cc:729] T 5e6a8fbaa28c40a89e0828e7af7fba25 P 7b751a5466824e6da229fbeb41963618 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b751a5466824e6da229fbeb41963618, State: Initialized, Role: FOLLOWER
I20260925 22:36:20.407300 11713 consensus_queue.cc:260] T 5e6a8fbaa28c40a89e0828e7af7fba25 P 7b751a5466824e6da229fbeb41963618 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b751a5466824e6da229fbeb41963618" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 32929 } } peers { permanent_uuid: "47690fe7d3d841c0b59693877c337108" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 42693 } } peers { permanent_uuid: "ef922b98e358410186dadab999b93698" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 39767 } }
I20260925 22:36:20.407881 11561 heartbeater.cc:503] Master 127.6.246.254:45707 was elected leader, sending a full tablet report...
I20260925 22:36:20.408155 11713 ts_tablet_manager.cc:1428] T 5e6a8fbaa28c40a89e0828e7af7fba25 P 7b751a5466824e6da229fbeb41963618: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260925 22:36:20.433547 11719 raft_consensus.cc:482] T 5e6a8fbaa28c40a89e0828e7af7fba25 P 7b751a5466824e6da229fbeb41963618 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:36:20.433743 11719 raft_consensus.cc:504] T 5e6a8fbaa28c40a89e0828e7af7fba25 P 7b751a5466824e6da229fbeb41963618 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b751a5466824e6da229fbeb41963618" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 32929 } } peers { permanent_uuid: "47690fe7d3d841c0b59693877c337108" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 42693 } } peers { permanent_uuid: "ef922b98e358410186dadab999b93698" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 39767 } }
I20260925 22:36:20.434828 11719 leader_election.cc:288] T 5e6a8fbaa28c40a89e0828e7af7fba25 P 7b751a5466824e6da229fbeb41963618 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47690fe7d3d841c0b59693877c337108 (127.6.246.195:42693), ef922b98e358410186dadab999b93698 (127.6.246.193:39767)
I20260925 22:36:20.438971 11563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:20.435158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.195:42693 (local address 127.6.246.194:34815)
0925 22:36:20.435282 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:20.435299 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:20.435407 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:20.435743 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:20.435746 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:20.435765 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:36:20.435911 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:20.435919 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:20.436857 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:20.436860 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:20.437765 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:20.437774 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:20.438542 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:20.438545 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:20.438650 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:20.438674 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 22:36:20.438757 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 22:36:20.438836 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 22:36:20.438979 11710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:20.435248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.194:34815 (local address 127.6.246.195:42693)
0925 22:36:20.435407 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:20.435413 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:20.435432 (+ 19us) server_negotiation.cc:407] Connection header received
0925 22:36:20.435509 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:20.435512 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:20.435559 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:20.435636 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:36:20.436031 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:20.436743 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:20.437896 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:20.438437 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:20.438527 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:20.438564 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:20.438775 (+ 211us) server_negotiation.cc:299] Negotiation successful
0925 22:36:20.438860 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 22:36:20.439327 11709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:20.435581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.194:47591 (local address 127.6.246.193:39767)
0925 22:36:20.435697 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:20.435701 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:20.435846 (+ 145us) server_negotiation.cc:407] Connection header received
0925 22:36:20.435933 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:20.435935 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:20.435999 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:20.436098 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:36:20.436385 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:20.437120 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:20.438254 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:20.438751 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:20.438852 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:20.438904 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:20.439134 (+ 230us) server_negotiation.cc:299] Negotiation successful
0925 22:36:20.439217 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 22:36:20.439654 11720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:20.435427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.193:39767 (local address 127.6.246.194:47591)
0925 22:36:20.435756 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:20.435768 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:20.435856 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:20.436112 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:20.436117 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:20.436136 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:36:20.436244 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:20.436253 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:20.437230 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:20.437233 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:20.438123 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:20.438134 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:20.438863 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:20.438866 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:20.438970 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:20.439000 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 22:36:20.439459 (+ 459us) client_negotiation.cc:240] Negotiation successful
0925 22:36:20.439527 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":157,"threads_started":1}
I20260925 22:36:20.439649 11648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e6a8fbaa28c40a89e0828e7af7fba25" candidate_uuid: "7b751a5466824e6da229fbeb41963618" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47690fe7d3d841c0b59693877c337108" is_pre_election: true
I20260925 22:36:20.439955 11648 raft_consensus.cc:2454] T 5e6a8fbaa28c40a89e0828e7af7fba25 P 47690fe7d3d841c0b59693877c337108 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b751a5466824e6da229fbeb41963618 in term 0.
I20260925 22:36:20.440212 11382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e6a8fbaa28c40a89e0828e7af7fba25" candidate_uuid: "7b751a5466824e6da229fbeb41963618" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef922b98e358410186dadab999b93698" is_pre_election: true
I20260925 22:36:20.440517 11382 raft_consensus.cc:2454] T 5e6a8fbaa28c40a89e0828e7af7fba25 P ef922b98e358410186dadab999b93698 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b751a5466824e6da229fbeb41963618 in term 0.
I20260925 22:36:20.440491 11448 leader_election.cc:302] T 5e6a8fbaa28c40a89e0828e7af7fba25 P 7b751a5466824e6da229fbeb41963618 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47690fe7d3d841c0b59693877c337108, 7b751a5466824e6da229fbeb41963618; no voters:
I20260925 22:36:20.440770 11719 raft_consensus.cc:2781] T 5e6a8fbaa28c40a89e0828e7af7fba25 P 7b751a5466824e6da229fbeb41963618 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:36:20.440856 11719 raft_consensus.cc:482] T 5e6a8fbaa28c40a89e0828e7af7fba25 P 7b751a5466824e6da229fbeb41963618 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:36:20.440925 11719 raft_consensus.cc:3037] T 5e6a8fbaa28c40a89e0828e7af7fba25 P 7b751a5466824e6da229fbeb41963618 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:36:20.442122 11719 raft_consensus.cc:504] T 5e6a8fbaa28c40a89e0828e7af7fba25 P 7b751a5466824e6da229fbeb41963618 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b751a5466824e6da229fbeb41963618" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 32929 } } peers { permanent_uuid: "47690fe7d3d841c0b59693877c337108" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 42693 } } peers { permanent_uuid: "ef922b98e358410186dadab999b93698" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 39767 } }
I20260925 22:36:20.442467 11719 leader_election.cc:288] T 5e6a8fbaa28c40a89e0828e7af7fba25 P 7b751a5466824e6da229fbeb41963618 [CANDIDATE]: Term 1 election: Requested vote from peers 47690fe7d3d841c0b59693877c337108 (127.6.246.195:42693), ef922b98e358410186dadab999b93698 (127.6.246.193:39767)
I20260925 22:36:20.442785 11648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e6a8fbaa28c40a89e0828e7af7fba25" candidate_uuid: "7b751a5466824e6da229fbeb41963618" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47690fe7d3d841c0b59693877c337108"
I20260925 22:36:20.442881 11382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e6a8fbaa28c40a89e0828e7af7fba25" candidate_uuid: "7b751a5466824e6da229fbeb41963618" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef922b98e358410186dadab999b93698"
I20260925 22:36:20.442921 11648 raft_consensus.cc:3037] T 5e6a8fbaa28c40a89e0828e7af7fba25 P 47690fe7d3d841c0b59693877c337108 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:36:20.443011 11382 raft_consensus.cc:3037] T 5e6a8fbaa28c40a89e0828e7af7fba25 P ef922b98e358410186dadab999b93698 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:36:20.443933 11648 raft_consensus.cc:2454] T 5e6a8fbaa28c40a89e0828e7af7fba25 P 47690fe7d3d841c0b59693877c337108 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b751a5466824e6da229fbeb41963618 in term 1.
I20260925 22:36:20.444087 11382 raft_consensus.cc:2454] T 5e6a8fbaa28c40a89e0828e7af7fba25 P ef922b98e358410186dadab999b93698 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b751a5466824e6da229fbeb41963618 in term 1.
I20260925 22:36:20.444217 11448 leader_election.cc:302] T 5e6a8fbaa28c40a89e0828e7af7fba25 P 7b751a5466824e6da229fbeb41963618 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47690fe7d3d841c0b59693877c337108, 7b751a5466824e6da229fbeb41963618; no voters:
I20260925 22:36:20.444433 11719 raft_consensus.cc:2781] T 5e6a8fbaa28c40a89e0828e7af7fba25 P 7b751a5466824e6da229fbeb41963618 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:36:20.444682 11719 raft_consensus.cc:686] T 5e6a8fbaa28c40a89e0828e7af7fba25 P 7b751a5466824e6da229fbeb41963618 [term 1 LEADER]: Becoming Leader. State: Replica: 7b751a5466824e6da229fbeb41963618, State: Running, Role: LEADER
I20260925 22:36:20.444988 11719 consensus_queue.cc:237] T 5e6a8fbaa28c40a89e0828e7af7fba25 P 7b751a5466824e6da229fbeb41963618 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b751a5466824e6da229fbeb41963618" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 32929 } } peers { permanent_uuid: "47690fe7d3d841c0b59693877c337108" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 42693 } } peers { permanent_uuid: "ef922b98e358410186dadab999b93698" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 39767 } }
I20260925 22:36:20.447793 11237 catalog_manager.cc:5495] T 5e6a8fbaa28c40a89e0828e7af7fba25 P 7b751a5466824e6da229fbeb41963618 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b751a5466824e6da229fbeb41963618 (127.6.246.194). New cstate: current_term: 1 leader_uuid: "7b751a5466824e6da229fbeb41963618" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b751a5466824e6da229fbeb41963618" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 32929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47690fe7d3d841c0b59693877c337108" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 42693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef922b98e358410186dadab999b93698" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 39767 } health_report { overall_health: UNKNOWN } } }
I20260925 22:36:20.460717 11707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:20.457336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37394 (local address 127.6.246.194:32929)
0925 22:36:20.457462 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:20.457465 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:20.457478 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:36:20.457594 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:20.457597 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:20.457648 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:20.457734 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:36:20.457954 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:20.458441 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:20.459359 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:20.459560 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:20.459634 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:20.459913 (+ 279us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:36:20.460307 (+ 394us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:36:20.460398 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:20.460548 (+ 150us) server_negotiation.cc:299] Negotiation successful
0925 22:36:20.460597 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 22:36:20.466545 11382 raft_consensus.cc:1261] T 5e6a8fbaa28c40a89e0828e7af7fba25 P ef922b98e358410186dadab999b93698 [term 1 FOLLOWER]: Refusing update from remote peer 7b751a5466824e6da229fbeb41963618: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:36:20.466545 11648 raft_consensus.cc:1261] T 5e6a8fbaa28c40a89e0828e7af7fba25 P 47690fe7d3d841c0b59693877c337108 [term 1 FOLLOWER]: Refusing update from remote peer 7b751a5466824e6da229fbeb41963618: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:36:20.467257 11719 consensus_queue.cc:1035] T 5e6a8fbaa28c40a89e0828e7af7fba25 P 7b751a5466824e6da229fbeb41963618 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47690fe7d3d841c0b59693877c337108" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 42693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:36:20.467439 11721 consensus_queue.cc:1035] T 5e6a8fbaa28c40a89e0828e7af7fba25 P 7b751a5466824e6da229fbeb41963618 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef922b98e358410186dadab999b93698" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 39767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:36:20.476226 11726 mvcc.cc:205] Tried to move back new op lower bound from 7333379196783403008 to 7333379196705271808. Current Snapshot: MvccSnapshot[applied={T|T < 7333379196783403008}]
I20260925 22:36:20.483352 11236 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:36:20.483542 11236 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:36:20.485984 11236 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 49ac656251a84d86be24cf9ea223666c: Sending DeleteTablet for 3 replicas of tablet 5e6a8fbaa28c40a89e0828e7af7fba25
I20260925 22:36:20.486732 11495 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e6a8fbaa28c40a89e0828e7af7fba25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:36:20 UTC) from {username='kudu', principal='kudu/127.6.246.254@KRBTEST.COM'} at 127.0.0.1:37378
I20260925 22:36:20.486792 11362 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e6a8fbaa28c40a89e0828e7af7fba25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:36:20 UTC) from {username='kudu', principal='kudu/127.6.246.254@KRBTEST.COM'} at 127.0.0.1:38864
I20260925 22:36:20.486815 11628 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e6a8fbaa28c40a89e0828e7af7fba25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:36:20 UTC) from {username='kudu', principal='kudu/127.6.246.254@KRBTEST.COM'} at 127.0.0.1:38450
I20260925 22:36:20.487224 11734 tablet_replica.cc:330] T 5e6a8fbaa28c40a89e0828e7af7fba25 P 7b751a5466824e6da229fbeb41963618: stopping tablet replica
I20260925 22:36:20.487386 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 11300
I20260925 22:36:20.488741 11734 raft_consensus.cc:2229] T 5e6a8fbaa28c40a89e0828e7af7fba25 P 7b751a5466824e6da229fbeb41963618 [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:36:20.488933 11735 tablet_replica.cc:330] T 5e6a8fbaa28c40a89e0828e7af7fba25 P 47690fe7d3d841c0b59693877c337108: stopping tablet replica
I20260925 22:36:20.489179 11734 raft_consensus.cc:2258] T 5e6a8fbaa28c40a89e0828e7af7fba25 P 7b751a5466824e6da229fbeb41963618 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:36:20.489241 11735 raft_consensus.cc:2229] T 5e6a8fbaa28c40a89e0828e7af7fba25 P 47690fe7d3d841c0b59693877c337108 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:36:20.489578 11735 raft_consensus.cc:2258] T 5e6a8fbaa28c40a89e0828e7af7fba25 P 47690fe7d3d841c0b59693877c337108 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:36:20.491127 11735 ts_tablet_manager.cc:1905] T 5e6a8fbaa28c40a89e0828e7af7fba25 P 47690fe7d3d841c0b59693877c337108: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:36:20.491472 11734 ts_tablet_manager.cc:1905] T 5e6a8fbaa28c40a89e0828e7af7fba25 P 7b751a5466824e6da229fbeb41963618: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 22:36:20.495124 11222 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.246.193:39767 (error 108)
I20260925 22:36:20.495846 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 11433
I20260925 22:36:20.496304 11706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:20.495915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.193:39767 (local address 127.0.0.1:38870)
0925 22:36:20.496111 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:20.496202 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.193:39767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
W20260925 22:36:20.496556 11222 catalog_manager.cc:4584] TS ef922b98e358410186dadab999b93698 (127.6.246.193:39767): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5e6a8fbaa28c40a89e0828e7af7fba25: Network error: Client connection negotiation failed: client connection to 127.6.246.193:39767: connect: Connection refused (error 111)
I20260925 22:36:20.513633 11735 ts_tablet_manager.cc:1918] T 5e6a8fbaa28c40a89e0828e7af7fba25 P 47690fe7d3d841c0b59693877c337108: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:36:20.513813 11735 log.cc:1196] T 5e6a8fbaa28c40a89e0828e7af7fba25 P 47690fe7d3d841c0b59693877c337108: Deleting WAL directory at /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790375752797504-7131-0/minicluster-data/ts-2/wal/wals/5e6a8fbaa28c40a89e0828e7af7fba25
I20260925 22:36:20.514173 11735 ts_tablet_manager.cc:1939] T 5e6a8fbaa28c40a89e0828e7af7fba25 P 47690fe7d3d841c0b59693877c337108: Deleting consensus metadata
I20260925 22:36:20.515554 11222 catalog_manager.cc:4841] TS 47690fe7d3d841c0b59693877c337108 (127.6.246.195:42693): tablet 5e6a8fbaa28c40a89e0828e7af7fba25 (table test-table [id=f28e1ca587f943feafa8ea355433ea5b]) successfully deleted
I20260925 22:36:20.518326 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 11566
I20260925 22:36:20.519160 11706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:20.518499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.194:32929 (local address 127.0.0.1:37408)
0925 22:36:20.518984 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:20.519049 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.194:32929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382}
I20260925 22:36:20.523813 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 11207
2026-09-25T22:36:20Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2230 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-25T22:36:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:36:20Z Disabled control of system clock
I20260925 22:36:20.553578 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790375752797504-7131-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790375752797504-7131-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790375752797504-7131-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:36639
--webserver_interface=127.6.246.254
--webserver_port=0
--builtin_ntp_servers=127.6.246.212:39243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.246.254:36639
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 22:36:20.661242 11740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:36:20.661499 11740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:36:20.661533 11740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:36:20.665467 11740 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:36:20.665516 11740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:36:20.665531 11740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:36:20.665548 11740 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:36:20.665563 11740 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:36:20.670274 11740 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:39243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790375752797504-7131-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790375752797504-7131-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.246.254:36639
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:36639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.246.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11740
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790375752797504-7131-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:36:20.671285 11740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:36:20.672672 11740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:36:20.680796 11748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:20.680830 11746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:20.680806 11745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:20.681113 11740 server_base.cc:1034] running on GCE node
I20260925 22:36:20.681749 11740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:36:20.682796 11740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:36:20.683990 11740 hybrid_clock.cc:648] HybridClock initialized: now 1790375780683965 us; error 41 us; skew 500 ppm
I20260925 22:36:20.691295 11740 webserver.cc:460] Webserver started at http://127.6.246.254:40779/ using document root <none> and password file <none>
I20260925 22:36:20.691992 11740 fs_manager.cc:359] Metadata directory not provided
I20260925 22:36:20.692080 11740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:36:20.692270 11740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:36:20.694342 11740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790375752797504-7131-0/minicluster-data/master-0/data/instance:
uuid: "8b5f50027f574cf9ac678befb1b1ddd5"
format_stamp: "Formatted at 2026-09-25 22:36:20 on dist-test-slave-wjdr"
server_key: "4965bb515a5149d42fbac684aeb0d807"
server_key_iv: "6a41acd5180d8aed052074475e5ac3f3"
server_key_version: "encryptionkey@0"
I20260925 22:36:20.694831 11740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790375752797504-7131-0/minicluster-data/master-0/wal/instance:
uuid: "8b5f50027f574cf9ac678befb1b1ddd5"
format_stamp: "Formatted at 2026-09-25 22:36:20 on dist-test-slave-wjdr"
server_key: "4965bb515a5149d42fbac684aeb0d807"
server_key_iv: "6a41acd5180d8aed052074475e5ac3f3"
server_key_version: "encryptionkey@0"
I20260925 22:36:20.698602 11740 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:36:20.701062 11754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:20.702471 11740 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:36:20.702571 11740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790375752797504-7131-0/minicluster-data/master-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790375752797504-7131-0/minicluster-data/master-0/wal
uuid: "8b5f50027f574cf9ac678befb1b1ddd5"
format_stamp: "Formatted at 2026-09-25 22:36:20 on dist-test-slave-wjdr"
server_key: "4965bb515a5149d42fbac684aeb0d807"
server_key_iv: "6a41acd5180d8aed052074475e5ac3f3"
server_key_version: "encryptionkey@0"
I20260925 22:36:20.702699 11740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790375752797504-7131-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790375752797504-7131-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790375752797504-7131-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:36:20.707023 11740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:36:20.707607 11740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:36:20.707791 11740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:36:20.714386 11740 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.254:36639
I20260925 22:36:20.714452 11806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.254:36639 every 8 connection(s)
I20260925 22:36:20.715281 11740 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
I20260925 22:36:20.718550 11807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:36:20.718725 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 11740
I20260925 22:36:20.718844 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790375752797504-7131-0/minicluster-data/master-0/wal/instance
I20260925 22:36:20.719157 7131 external_mini_cluster.cc:1428] Setting key 634f917b707b63fe0590ecae849af22d
I20260925 22:36:20.725144 11810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:20.720693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45166 (local address 127.6.246.254:36639)
0925 22:36:20.721335 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:20.721354 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:20.721396 (+ 42us) server_negotiation.cc:407] Connection header received
0925 22:36:20.722157 (+ 761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:20.722181 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:20.722615 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:20.722974 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:20.723676 (+ 702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:20.723681 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:20.723695 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:36:20.723706 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:20.723767 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:36:20.723789 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:20.723925 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:20.724023 (+ 98us) server_negotiation.cc:299] Negotiation successful
0925 22:36:20.724258 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":102,"threads_started":1}
I20260925 22:36:20.725996 11807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b5f50027f574cf9ac678befb1b1ddd5: Bootstrap starting.
I20260925 22:36:20.728174 11807 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b5f50027f574cf9ac678befb1b1ddd5: Neither blocks nor log segments found. Creating new log.
I20260925 22:36:20.728941 11807 log.cc:824] T 00000000000000000000000000000000 P 8b5f50027f574cf9ac678befb1b1ddd5: Log is configured to *not* fsync() on all Append() calls
I20260925 22:36:20.731312 11807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b5f50027f574cf9ac678befb1b1ddd5: No bootstrap required, opened a new log
I20260925 22:36:20.734944 11807 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b5f50027f574cf9ac678befb1b1ddd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b5f50027f574cf9ac678befb1b1ddd5" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 36639 } }
I20260925 22:36:20.735145 11807 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b5f50027f574cf9ac678befb1b1ddd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:36:20.735204 11807 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b5f50027f574cf9ac678befb1b1ddd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b5f50027f574cf9ac678befb1b1ddd5, State: Initialized, Role: FOLLOWER
I20260925 22:36:20.735594 11807 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b5f50027f574cf9ac678befb1b1ddd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b5f50027f574cf9ac678befb1b1ddd5" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 36639 } }
I20260925 22:36:20.735708 11807 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b5f50027f574cf9ac678befb1b1ddd5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:36:20.735800 11807 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b5f50027f574cf9ac678befb1b1ddd5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:36:20.735903 11807 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b5f50027f574cf9ac678befb1b1ddd5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:36:20.736833 11807 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b5f50027f574cf9ac678befb1b1ddd5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b5f50027f574cf9ac678befb1b1ddd5" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 36639 } }
I20260925 22:36:20.737160 11807 leader_election.cc:302] T 00000000000000000000000000000000 P 8b5f50027f574cf9ac678befb1b1ddd5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b5f50027f574cf9ac678befb1b1ddd5; no voters:
I20260925 22:36:20.737443 11807 leader_election.cc:288] T 00000000000000000000000000000000 P 8b5f50027f574cf9ac678befb1b1ddd5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:36:20.737557 11812 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b5f50027f574cf9ac678befb1b1ddd5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:36:20.737802 11812 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b5f50027f574cf9ac678befb1b1ddd5 [term 1 LEADER]: Becoming Leader. State: Replica: 8b5f50027f574cf9ac678befb1b1ddd5, State: Running, Role: LEADER
I20260925 22:36:20.738095 11812 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b5f50027f574cf9ac678befb1b1ddd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b5f50027f574cf9ac678befb1b1ddd5" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 36639 } }
I20260925 22:36:20.738642 11807 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b5f50027f574cf9ac678befb1b1ddd5 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:36:20.739225 11814 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b5f50027f574cf9ac678befb1b1ddd5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b5f50027f574cf9ac678befb1b1ddd5. Latest consensus state: current_term: 1 leader_uuid: "8b5f50027f574cf9ac678befb1b1ddd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b5f50027f574cf9ac678befb1b1ddd5" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 36639 } } }
I20260925 22:36:20.739389 11814 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b5f50027f574cf9ac678befb1b1ddd5 [sys.catalog]: This master's current role is: LEADER
I20260925 22:36:20.739650 11813 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b5f50027f574cf9ac678befb1b1ddd5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b5f50027f574cf9ac678befb1b1ddd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b5f50027f574cf9ac678befb1b1ddd5" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 36639 } } }
I20260925 22:36:20.739759 11813 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b5f50027f574cf9ac678befb1b1ddd5 [sys.catalog]: This master's current role is: LEADER
W20260925 22:36:20.743666 11828 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8b5f50027f574cf9ac678befb1b1ddd5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:36:20.743757 11828 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:36:20.743840 11820 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:36:20.744586 11820 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:36:20.749281 11820 catalog_manager.cc:1347] Generated new cluster ID: 71e341da25a9487fa295cdffa8aa1241
I20260925 22:36:20.749346 11820 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:36:20.759191 11820 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:36:20.759493 11820 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:36:20.777547 11820 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b5f50027f574cf9ac678befb1b1ddd5: Generated new TSK 0
I20260925 22:36:20.778370 11820 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:36:20.791615 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790375752797504-7131-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790375752797504-7131-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790375752797504-7131-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.193:0
--local_ip_for_outbound_sockets=127.6.246.193
--webserver_interface=127.6.246.193
--webserver_port=0
--tserver_master_addrs=127.6.246.254:36639
--builtin_ntp_servers=127.6.246.212:39243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 22:36:20.910202 11831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:36:20.910514 11831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:36:20.910594 11831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:36:20.915058 11831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:36:20.915143 11831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:36:20.915232 11831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.193
I20260925 22:36:20.920068 11831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:39243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790375752797504-7131-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790375752797504-7131-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.246.193
--webserver_port=0
--tserver_master_addrs=127.6.246.254:36639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11831
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.193
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790375752797504-7131-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:36:20.921133 11831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:36:20.922525 11831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:36:20.932488 11836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:20.932644 11839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:20.932494 11837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:20.932876 11831 server_base.cc:1034] running on GCE node
I20260925 22:36:20.933318 11831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:36:20.933930 11831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:36:20.935156 11831 hybrid_clock.cc:648] HybridClock initialized: now 1790375780935136 us; error 31 us; skew 500 ppm
I20260925 22:36:20.937089 11831 webserver.cc:460] Webserver started at http://127.6.246.193:36959/ using document root <none> and password file <none>
I20260925 22:36:20.937701 11831 fs_manager.cc:359] Metadata directory not provided
I20260925 22:36:20.937767 11831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:36:20.937930 11831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:36:20.940088 11831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790375752797504-7131-0/minicluster-data/ts-0/data/instance:
uuid: "100e75ff3c2a447491843e27e50516eb"
format_stamp: "Formatted at 2026-09-25 22:36:20 on dist-test-slave-wjdr"
server_key: "6e4e4c6c93eba12bfdaebbc8612b3aa3"
server_key_iv: "19e0ef6ad0294f20617f91a678b8d015"
server_key_version: "encryptionkey@0"
I20260925 22:36:20.940650 11831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790375752797504-7131-0/minicluster-data/ts-0/wal/instance:
uuid: "100e75ff3c2a447491843e27e50516eb"
format_stamp: "Formatted at 2026-09-25 22:36:20 on dist-test-slave-wjdr"
server_key: "6e4e4c6c93eba12bfdaebbc8612b3aa3"
server_key_iv: "19e0ef6ad0294f20617f91a678b8d015"
server_key_version: "encryptionkey@0"
I20260925 22:36:20.944180 11831 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:36:20.946648 11845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:20.948071 11831 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:36:20.948167 11831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790375752797504-7131-0/minicluster-data/ts-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790375752797504-7131-0/minicluster-data/ts-0/wal
uuid: "100e75ff3c2a447491843e27e50516eb"
format_stamp: "Formatted at 2026-09-25 22:36:20 on dist-test-slave-wjdr"
server_key: "6e4e4c6c93eba12bfdaebbc8612b3aa3"
server_key_iv: "19e0ef6ad0294f20617f91a678b8d015"
server_key_version: "encryptionkey@0"
I20260925 22:36:20.948261 11831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790375752797504-7131-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790375752797504-7131-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790375752797504-7131-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:36:20.953238 11831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:36:20.953814 11831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:36:20.953955 11831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:36:20.954700 11831 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:36:20.955595 11831 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:36:20.955642 11831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:20.955679 11831 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:36:20.955694 11831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:20.964919 11831 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.193:38659
I20260925 22:36:20.964946 11958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.193:38659 every 8 connection(s)
I20260925 22:36:20.965762 11831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
I20260925 22:36:20.968129 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 11831
I20260925 22:36:20.968273 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790375752797504-7131-0/minicluster-data/ts-0/wal/instance
I20260925 22:36:20.968623 7131 external_mini_cluster.cc:1428] Setting key 44646646b9c18b01d78491e24b011089
I20260925 22:36:20.970741 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790375752797504-7131-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790375752797504-7131-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790375752797504-7131-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.194:0
--local_ip_for_outbound_sockets=127.6.246.194
--webserver_interface=127.6.246.194
--webserver_port=0
--tserver_master_addrs=127.6.246.254:36639
--builtin_ntp_servers=127.6.246.212:39243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260925 22:36:20.972376 11810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:20.967532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.193:40005 (local address 127.6.246.254:36639)
0925 22:36:20.967662 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:20.967665 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:20.968937 (+ 1272us) server_negotiation.cc:407] Connection header received
0925 22:36:20.970401 (+ 1464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:20.970404 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:20.970445 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:20.970546 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:20.971755 (+ 1209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:20.971762 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:20.971764 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:36:20.971776 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:20.971801 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:36:20.971813 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:20.971910 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:20.972148 (+ 238us) server_negotiation.cc:299] Negotiation successful
0925 22:36:20.972199 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 22:36:20.973292 11961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:20.967923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:36639 (local address 127.6.246.193:40005)
0925 22:36:20.968754 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:20.968835 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:20.970197 (+ 1362us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:20.970711 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:20.970717 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:20.970829 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:20.971496 (+ 667us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:36:20.971498 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:20.971539 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:36:20.971607 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:36:20.971941 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:36:20.971956 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:36:20.971967 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 22:36:20.972172 (+ 205us) client_negotiation.cc:240] Negotiation successful
0925 22:36:20.972374 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":146,"threads_started":1}
I20260925 22:36:20.974599 11959 heartbeater.cc:347] Connected to a master server at 127.6.246.254:36639
I20260925 22:36:20.974849 11959 heartbeater.cc:464] Registering TS with master...
I20260925 22:36:20.975348 11959 heartbeater.cc:511] Master 127.6.246.254:36639 requested a full tablet report, sending...
I20260925 22:36:20.977072 11767 ts_manager.cc:194] Registered new tserver with Master: 100e75ff3c2a447491843e27e50516eb (127.6.246.193:38659)
W20260925 22:36:21.084335 11962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:36:21.084592 11962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:36:21.084661 11962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:36:21.088837 11962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:36:21.088889 11962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:36:21.088979 11962 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.194
I20260925 22:36:21.093997 11962 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:39243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790375752797504-7131-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790375752797504-7131-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.246.194
--webserver_port=0
--tserver_master_addrs=127.6.246.254:36639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.194
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790375752797504-7131-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:36:21.095077 11962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:36:21.096436 11962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:36:21.105775 11967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:21.105799 11970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:21.105850 11962 server_base.cc:1034] running on GCE node
W20260925 22:36:21.105790 11968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:21.106333 11962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:36:21.107012 11962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:36:21.108203 11962 hybrid_clock.cc:648] HybridClock initialized: now 1790375781108191 us; error 26 us; skew 500 ppm
I20260925 22:36:21.110131 11962 webserver.cc:460] Webserver started at http://127.6.246.194:41969/ using document root <none> and password file <none>
I20260925 22:36:21.110708 11962 fs_manager.cc:359] Metadata directory not provided
I20260925 22:36:21.110800 11962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:36:21.111013 11962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:36:21.113070 11962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790375752797504-7131-0/minicluster-data/ts-1/data/instance:
uuid: "95fe301fe790464995b14857f26307c0"
format_stamp: "Formatted at 2026-09-25 22:36:21 on dist-test-slave-wjdr"
server_key: "d56de3fdb5caedc0059e38d901c0dc0b"
server_key_iv: "c41fae6b6fc9430645bc23192b7194ef"
server_key_version: "encryptionkey@0"
I20260925 22:36:21.113637 11962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790375752797504-7131-0/minicluster-data/ts-1/wal/instance:
uuid: "95fe301fe790464995b14857f26307c0"
format_stamp: "Formatted at 2026-09-25 22:36:21 on dist-test-slave-wjdr"
server_key: "d56de3fdb5caedc0059e38d901c0dc0b"
server_key_iv: "c41fae6b6fc9430645bc23192b7194ef"
server_key_version: "encryptionkey@0"
I20260925 22:36:21.117208 11962 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:36:21.119596 11976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:21.120915 11962 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:36:21.121044 11962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790375752797504-7131-0/minicluster-data/ts-1/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790375752797504-7131-0/minicluster-data/ts-1/wal
uuid: "95fe301fe790464995b14857f26307c0"
format_stamp: "Formatted at 2026-09-25 22:36:21 on dist-test-slave-wjdr"
server_key: "d56de3fdb5caedc0059e38d901c0dc0b"
server_key_iv: "c41fae6b6fc9430645bc23192b7194ef"
server_key_version: "encryptionkey@0"
I20260925 22:36:21.121153 11962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790375752797504-7131-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790375752797504-7131-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790375752797504-7131-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:36:21.125627 11962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:36:21.126194 11962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:36:21.126368 11962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:36:21.126969 11962 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:36:21.127851 11962 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:36:21.127925 11962 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:21.127990 11962 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:36:21.128042 11962 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:21.136793 11962 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.194:46403
I20260925 22:36:21.136871 12089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.194:46403 every 8 connection(s)
I20260925 22:36:21.137748 11962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
I20260925 22:36:21.143731 11810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:21.139703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.194:60643 (local address 127.6.246.254:36639)
0925 22:36:21.139851 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:21.139856 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:21.141025 (+ 1169us) server_negotiation.cc:407] Connection header received
0925 22:36:21.142269 (+ 1244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:21.142272 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:21.142309 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:21.142385 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:21.143367 (+ 982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:21.143370 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:21.143372 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:36:21.143385 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:21.143404 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:36:21.143412 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:21.143485 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:21.143584 (+ 99us) server_negotiation.cc:299] Negotiation successful
0925 22:36:21.143621 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 22:36:21.144650 12092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:21.140156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:36639 (local address 127.6.246.194:60643)
0925 22:36:21.140882 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:21.140923 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:21.142082 (+ 1159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:21.142526 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:21.142532 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:21.142621 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:21.143131 (+ 510us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:36:21.143132 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:21.143170 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:36:21.143233 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:36:21.143469 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:36:21.143473 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:36:21.143477 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 22:36:21.143592 (+ 115us) client_negotiation.cc:240] Negotiation successful
0925 22:36:21.143807 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":108,"threads_started":1}
I20260925 22:36:21.145880 12090 heartbeater.cc:347] Connected to a master server at 127.6.246.254:36639
I20260925 22:36:21.146107 12090 heartbeater.cc:464] Registering TS with master...
I20260925 22:36:21.146503 12090 heartbeater.cc:511] Master 127.6.246.254:36639 requested a full tablet report, sending...
I20260925 22:36:21.147239 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 11962
I20260925 22:36:21.147392 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790375752797504-7131-0/minicluster-data/ts-1/wal/instance
I20260925 22:36:21.147517 11767 ts_manager.cc:194] Registered new tserver with Master: 95fe301fe790464995b14857f26307c0 (127.6.246.194:46403)
I20260925 22:36:21.147708 7131 external_mini_cluster.cc:1428] Setting key ff47c9d79fe0c7ea2fb412f32beaf621
I20260925 22:36:21.149582 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790375752797504-7131-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790375752797504-7131-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790375752797504-7131-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.195:0
--local_ip_for_outbound_sockets=127.6.246.195
--webserver_interface=127.6.246.195
--webserver_port=0
--tserver_master_addrs=127.6.246.254:36639
--builtin_ntp_servers=127.6.246.212:39243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 22:36:21.259322 12093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:36:21.259610 12093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:36:21.259676 12093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:36:21.263758 12093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:36:21.263844 12093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:36:21.263929 12093 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.195
I20260925 22:36:21.268756 12093 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:39243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790375752797504-7131-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790375752797504-7131-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.246.195
--webserver_port=0
--tserver_master_addrs=127.6.246.254:36639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12093
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.195
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790375752797504-7131-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:36:21.269812 12093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:36:21.271145 12093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:36:21.280356 12099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:21.280364 12101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:21.280536 12093 server_base.cc:1034] running on GCE node
W20260925 22:36:21.280556 12098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:21.281095 12093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:36:21.281675 12093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:36:21.282876 12093 hybrid_clock.cc:648] HybridClock initialized: now 1790375781282864 us; error 25 us; skew 500 ppm
I20260925 22:36:21.284999 12093 webserver.cc:460] Webserver started at http://127.6.246.195:39799/ using document root <none> and password file <none>
I20260925 22:36:21.285558 12093 fs_manager.cc:359] Metadata directory not provided
I20260925 22:36:21.285619 12093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:36:21.285826 12093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:36:21.287839 12093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790375752797504-7131-0/minicluster-data/ts-2/data/instance:
uuid: "8cdeb04a5f7c4122bc0faea6e4b57b85"
format_stamp: "Formatted at 2026-09-25 22:36:21 on dist-test-slave-wjdr"
server_key: "65313cc55843b2e5d7b142b6ae5807ba"
server_key_iv: "3a79c7244d08dfe3e3924bcaa54c0f89"
server_key_version: "encryptionkey@0"
I20260925 22:36:21.288419 12093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790375752797504-7131-0/minicluster-data/ts-2/wal/instance:
uuid: "8cdeb04a5f7c4122bc0faea6e4b57b85"
format_stamp: "Formatted at 2026-09-25 22:36:21 on dist-test-slave-wjdr"
server_key: "65313cc55843b2e5d7b142b6ae5807ba"
server_key_iv: "3a79c7244d08dfe3e3924bcaa54c0f89"
server_key_version: "encryptionkey@0"
I20260925 22:36:21.292059 12093 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:36:21.294605 12107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:21.295876 12093 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 22:36:21.295994 12093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790375752797504-7131-0/minicluster-data/ts-2/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790375752797504-7131-0/minicluster-data/ts-2/wal
uuid: "8cdeb04a5f7c4122bc0faea6e4b57b85"
format_stamp: "Formatted at 2026-09-25 22:36:21 on dist-test-slave-wjdr"
server_key: "65313cc55843b2e5d7b142b6ae5807ba"
server_key_iv: "3a79c7244d08dfe3e3924bcaa54c0f89"
server_key_version: "encryptionkey@0"
I20260925 22:36:21.296101 12093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790375752797504-7131-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790375752797504-7131-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790375752797504-7131-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:36:21.300760 12093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:36:21.301338 12093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:36:21.301509 12093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:36:21.302076 12093 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:36:21.302930 12093 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:36:21.302975 12093 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:21.303041 12093 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:36:21.303083 12093 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:21.311872 12093 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.195:39161
I20260925 22:36:21.311898 12220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.195:39161 every 8 connection(s)
I20260925 22:36:21.312803 12093 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
I20260925 22:36:21.315492 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 12093
I20260925 22:36:21.315608 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790375752797504-7131-0/minicluster-data/ts-2/wal/instance
I20260925 22:36:21.315882 7131 external_mini_cluster.cc:1428] Setting key 4f1b16ef726998cffd9b689c84722d90
I20260925 22:36:21.319010 11810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:21.314521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.195:34163 (local address 127.6.246.254:36639)
0925 22:36:21.314658 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:21.314661 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:21.315667 (+ 1006us) server_negotiation.cc:407] Connection header received
0925 22:36:21.317361 (+ 1694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:21.317363 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:21.317397 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:21.317463 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:21.318615 (+ 1152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:21.318618 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:21.318620 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:36:21.318630 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:21.318649 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:36:21.318656 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:21.318724 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:21.318866 (+ 142us) server_negotiation.cc:299] Negotiation successful
0925 22:36:21.318904 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 22:36:21.319828 12223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:21.314904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:36639 (local address 127.6.246.195:34163)
0925 22:36:21.315504 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:21.315540 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:21.317139 (+ 1599us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:21.317629 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:21.317638 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:21.317748 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:21.318381 (+ 633us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:36:21.318382 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:21.318420 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:36:21.318485 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:36:21.318732 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:36:21.318737 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:36:21.318741 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 22:36:21.318895 (+ 154us) client_negotiation.cc:240] Negotiation successful
0925 22:36:21.319108 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":105,"threads_started":1}
I20260925 22:36:21.320888 12221 heartbeater.cc:347] Connected to a master server at 127.6.246.254:36639
I20260925 22:36:21.321100 12221 heartbeater.cc:464] Registering TS with master...
I20260925 22:36:21.321465 12221 heartbeater.cc:511] Master 127.6.246.254:36639 requested a full tablet report, sending...
I20260925 22:36:21.322474 11767 ts_manager.cc:194] Registered new tserver with Master: 8cdeb04a5f7c4122bc0faea6e4b57b85 (127.6.246.195:39161)
I20260925 22:36:21.329664 7131 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:36:21.333093 11810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:21.331496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45180 (local address 127.6.246.254:36639)
0925 22:36:21.331621 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:21.331625 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:21.331905 (+ 280us) server_negotiation.cc:407] Connection header received
0925 22:36:21.331952 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:21.331954 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:21.331984 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:21.332062 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:21.332800 (+ 738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:21.332803 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:21.332804 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:36:21.332815 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:21.332832 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:36:21.332840 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:21.332921 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:21.332966 (+ 45us) server_negotiation.cc:299] Negotiation successful
0925 22:36:21.332996 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 22:36:21.335826 11767 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45180:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:36:21.339051 11767 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:36:21.350796 12236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:21.348205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44096 (local address 127.6.246.194:46403)
0925 22:36:21.348795 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:21.348807 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:21.348830 (+ 23us) server_negotiation.cc:407] Connection header received
0925 22:36:21.348881 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:21.348889 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:21.349049 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:21.349186 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:21.350091 (+ 905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:21.350097 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:21.350109 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:36:21.350120 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:21.350164 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:36:21.350195 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:21.350322 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:21.350475 (+ 153us) server_negotiation.cc:299] Negotiation successful
0925 22:36:21.350562 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":131,"threads_started":1}
I20260925 22:36:21.350888 12234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:21.347495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.193:38659 (local address 127.0.0.1:57528)
0925 22:36:21.348051 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:21.348101 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:21.348193 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:21.348987 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:21.348994 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:21.349142 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:21.349815 (+ 673us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:36:21.349818 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:21.349837 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:36:21.349905 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:36:21.350587 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:36:21.350592 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:36:21.350596 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 22:36:21.350677 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 22:36:21.350727 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":73,"threads_started":1}
I20260925 22:36:21.350970 12235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:21.347688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.194:46403 (local address 127.0.0.1:44096)
0925 22:36:21.348412 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:21.348425 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:21.348516 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:21.349183 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:21.349187 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:21.349201 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:21.349918 (+ 717us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:36:21.349920 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:21.349931 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:36:21.349959 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:36:21.350298 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:36:21.350309 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:36:21.350315 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 22:36:21.350449 (+ 134us) client_negotiation.cc:240] Negotiation successful
0925 22:36:21.350756 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"thread_start_us":124,"threads_started":1}
I20260925 22:36:21.351712 12233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:21.347657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57528 (local address 127.6.246.193:38659)
0925 22:36:21.348435 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:21.348448 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:21.348495 (+ 47us) server_negotiation.cc:407] Connection header received
0925 22:36:21.348562 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:21.348567 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:21.348816 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:21.349428 (+ 612us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:21.350328 (+ 900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:21.350336 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:21.350348 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:36:21.350358 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:21.350398 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:36:21.350443 (+ 45us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:21.350594 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:21.351452 (+ 858us) server_negotiation.cc:299] Negotiation successful
0925 22:36:21.351539 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":117,"threads_started":1}
I20260925 22:36:21.351879 12232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:21.348177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.195:39161 (local address 127.0.0.1:45506)
0925 22:36:21.349221 (+ 1044us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:21.349235 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:21.349310 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:21.350340 (+ 1030us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:21.350342 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:21.350356 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:21.350789 (+ 433us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:36:21.350793 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:21.350805 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:36:21.350836 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:36:21.351629 (+ 793us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:36:21.351632 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:36:21.351635 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 22:36:21.351711 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 22:36:21.351763 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":987,"thread_start_us":104,"threads_started":1}
I20260925 22:36:21.352311 12237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:21.348631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45506 (local address 127.6.246.195:39161)
0925 22:36:21.349077 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:21.349088 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:21.349408 (+ 320us) server_negotiation.cc:407] Connection header received
0925 22:36:21.349465 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:21.349472 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:21.349630 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:21.349754 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:21.350978 (+ 1224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:21.350984 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:21.350993 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:36:21.351006 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:21.351046 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:36:21.351074 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:21.351587 (+ 513us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:21.352046 (+ 459us) server_negotiation.cc:299] Negotiation successful
0925 22:36:21.352131 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":153,"threads_started":1}
I20260925 22:36:21.354637 11893 tablet_service.cc:1467] Processing CreateTablet for tablet 821633a5289948d1accbd91298ff0c0f (DEFAULT_TABLE table=test-table [id=eb9308ed3fc0433aaa93b0d3afc73108]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:36:21.354637 12024 tablet_service.cc:1467] Processing CreateTablet for tablet 821633a5289948d1accbd91298ff0c0f (DEFAULT_TABLE table=test-table [id=eb9308ed3fc0433aaa93b0d3afc73108]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:36:21.354637 12155 tablet_service.cc:1467] Processing CreateTablet for tablet 821633a5289948d1accbd91298ff0c0f (DEFAULT_TABLE table=test-table [id=eb9308ed3fc0433aaa93b0d3afc73108]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:36:21.355659 12155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 821633a5289948d1accbd91298ff0c0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:36:21.355687 12024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 821633a5289948d1accbd91298ff0c0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:36:21.355686 11893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 821633a5289948d1accbd91298ff0c0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:36:21.361516 12238 tablet_bootstrap.cc:492] T 821633a5289948d1accbd91298ff0c0f P 8cdeb04a5f7c4122bc0faea6e4b57b85: Bootstrap starting.
I20260925 22:36:21.363109 12239 tablet_bootstrap.cc:492] T 821633a5289948d1accbd91298ff0c0f P 100e75ff3c2a447491843e27e50516eb: Bootstrap starting.
I20260925 22:36:21.363109 12240 tablet_bootstrap.cc:492] T 821633a5289948d1accbd91298ff0c0f P 95fe301fe790464995b14857f26307c0: Bootstrap starting.
I20260925 22:36:21.363973 12238 tablet_bootstrap.cc:654] T 821633a5289948d1accbd91298ff0c0f P 8cdeb04a5f7c4122bc0faea6e4b57b85: Neither blocks nor log segments found. Creating new log.
I20260925 22:36:21.364794 12238 log.cc:824] T 821633a5289948d1accbd91298ff0c0f P 8cdeb04a5f7c4122bc0faea6e4b57b85: Log is configured to *not* fsync() on all Append() calls
I20260925 22:36:21.365235 12240 tablet_bootstrap.cc:654] T 821633a5289948d1accbd91298ff0c0f P 95fe301fe790464995b14857f26307c0: Neither blocks nor log segments found. Creating new log.
I20260925 22:36:21.365235 12239 tablet_bootstrap.cc:654] T 821633a5289948d1accbd91298ff0c0f P 100e75ff3c2a447491843e27e50516eb: Neither blocks nor log segments found. Creating new log.
I20260925 22:36:21.366065 12240 log.cc:824] T 821633a5289948d1accbd91298ff0c0f P 95fe301fe790464995b14857f26307c0: Log is configured to *not* fsync() on all Append() calls
I20260925 22:36:21.366065 12239 log.cc:824] T 821633a5289948d1accbd91298ff0c0f P 100e75ff3c2a447491843e27e50516eb: Log is configured to *not* fsync() on all Append() calls
I20260925 22:36:21.366700 12238 tablet_bootstrap.cc:492] T 821633a5289948d1accbd91298ff0c0f P 8cdeb04a5f7c4122bc0faea6e4b57b85: No bootstrap required, opened a new log
I20260925 22:36:21.366909 12238 ts_tablet_manager.cc:1397] T 821633a5289948d1accbd91298ff0c0f P 8cdeb04a5f7c4122bc0faea6e4b57b85: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 22:36:21.367892 12240 tablet_bootstrap.cc:492] T 821633a5289948d1accbd91298ff0c0f P 95fe301fe790464995b14857f26307c0: No bootstrap required, opened a new log
I20260925 22:36:21.367892 12239 tablet_bootstrap.cc:492] T 821633a5289948d1accbd91298ff0c0f P 100e75ff3c2a447491843e27e50516eb: No bootstrap required, opened a new log
I20260925 22:36:21.368065 12239 ts_tablet_manager.cc:1397] T 821633a5289948d1accbd91298ff0c0f P 100e75ff3c2a447491843e27e50516eb: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 22:36:21.368065 12240 ts_tablet_manager.cc:1397] T 821633a5289948d1accbd91298ff0c0f P 95fe301fe790464995b14857f26307c0: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 22:36:21.370584 12238 raft_consensus.cc:348] T 821633a5289948d1accbd91298ff0c0f P 8cdeb04a5f7c4122bc0faea6e4b57b85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100e75ff3c2a447491843e27e50516eb" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 38659 } } peers { permanent_uuid: "95fe301fe790464995b14857f26307c0" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 46403 } } peers { permanent_uuid: "8cdeb04a5f7c4122bc0faea6e4b57b85" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 39161 } }
I20260925 22:36:21.370818 12238 raft_consensus.cc:374] T 821633a5289948d1accbd91298ff0c0f P 8cdeb04a5f7c4122bc0faea6e4b57b85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:36:21.370888 12238 raft_consensus.cc:729] T 821633a5289948d1accbd91298ff0c0f P 8cdeb04a5f7c4122bc0faea6e4b57b85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cdeb04a5f7c4122bc0faea6e4b57b85, State: Initialized, Role: FOLLOWER
I20260925 22:36:21.371338 12238 consensus_queue.cc:260] T 821633a5289948d1accbd91298ff0c0f P 8cdeb04a5f7c4122bc0faea6e4b57b85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100e75ff3c2a447491843e27e50516eb" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 38659 } } peers { permanent_uuid: "95fe301fe790464995b14857f26307c0" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 46403 } } peers { permanent_uuid: "8cdeb04a5f7c4122bc0faea6e4b57b85" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 39161 } }
I20260925 22:36:21.372012 12221 heartbeater.cc:503] Master 127.6.246.254:36639 was elected leader, sending a full tablet report...
I20260925 22:36:21.372328 12238 ts_tablet_manager.cc:1428] T 821633a5289948d1accbd91298ff0c0f P 8cdeb04a5f7c4122bc0faea6e4b57b85: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260925 22:36:21.372584 12240 raft_consensus.cc:348] T 821633a5289948d1accbd91298ff0c0f P 95fe301fe790464995b14857f26307c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100e75ff3c2a447491843e27e50516eb" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 38659 } } peers { permanent_uuid: "95fe301fe790464995b14857f26307c0" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 46403 } } peers { permanent_uuid: "8cdeb04a5f7c4122bc0faea6e4b57b85" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 39161 } }
I20260925 22:36:21.372833 12240 raft_consensus.cc:374] T 821633a5289948d1accbd91298ff0c0f P 95fe301fe790464995b14857f26307c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:36:21.372888 12240 raft_consensus.cc:729] T 821633a5289948d1accbd91298ff0c0f P 95fe301fe790464995b14857f26307c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95fe301fe790464995b14857f26307c0, State: Initialized, Role: FOLLOWER
I20260925 22:36:21.373322 12240 consensus_queue.cc:260] T 821633a5289948d1accbd91298ff0c0f P 95fe301fe790464995b14857f26307c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100e75ff3c2a447491843e27e50516eb" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 38659 } } peers { permanent_uuid: "95fe301fe790464995b14857f26307c0" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 46403 } } peers { permanent_uuid: "8cdeb04a5f7c4122bc0faea6e4b57b85" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 39161 } }
I20260925 22:36:21.373257 12239 raft_consensus.cc:348] T 821633a5289948d1accbd91298ff0c0f P 100e75ff3c2a447491843e27e50516eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100e75ff3c2a447491843e27e50516eb" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 38659 } } peers { permanent_uuid: "95fe301fe790464995b14857f26307c0" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 46403 } } peers { permanent_uuid: "8cdeb04a5f7c4122bc0faea6e4b57b85" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 39161 } }
I20260925 22:36:21.373507 12239 raft_consensus.cc:374] T 821633a5289948d1accbd91298ff0c0f P 100e75ff3c2a447491843e27e50516eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:36:21.373566 12239 raft_consensus.cc:729] T 821633a5289948d1accbd91298ff0c0f P 100e75ff3c2a447491843e27e50516eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 100e75ff3c2a447491843e27e50516eb, State: Initialized, Role: FOLLOWER
I20260925 22:36:21.373996 12090 heartbeater.cc:503] Master 127.6.246.254:36639 was elected leader, sending a full tablet report...
I20260925 22:36:21.374075 12239 consensus_queue.cc:260] T 821633a5289948d1accbd91298ff0c0f P 100e75ff3c2a447491843e27e50516eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100e75ff3c2a447491843e27e50516eb" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 38659 } } peers { permanent_uuid: "95fe301fe790464995b14857f26307c0" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 46403 } } peers { permanent_uuid: "8cdeb04a5f7c4122bc0faea6e4b57b85" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 39161 } }
I20260925 22:36:21.374680 11959 heartbeater.cc:503] Master 127.6.246.254:36639 was elected leader, sending a full tablet report...
I20260925 22:36:21.375463 12240 ts_tablet_manager.cc:1428] T 821633a5289948d1accbd91298ff0c0f P 95fe301fe790464995b14857f26307c0: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260925 22:36:21.375022 12239 ts_tablet_manager.cc:1428] T 821633a5289948d1accbd91298ff0c0f P 100e75ff3c2a447491843e27e50516eb: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
W20260925 22:36:21.388744 12091 tablet.cc:2369] T 821633a5289948d1accbd91298ff0c0f P 95fe301fe790464995b14857f26307c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 22:36:21.466920 11960 tablet.cc:2369] T 821633a5289948d1accbd91298ff0c0f P 100e75ff3c2a447491843e27e50516eb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:36:21.503494 12246 raft_consensus.cc:482] T 821633a5289948d1accbd91298ff0c0f P 100e75ff3c2a447491843e27e50516eb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:36:21.503713 12246 raft_consensus.cc:504] T 821633a5289948d1accbd91298ff0c0f P 100e75ff3c2a447491843e27e50516eb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100e75ff3c2a447491843e27e50516eb" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 38659 } } peers { permanent_uuid: "95fe301fe790464995b14857f26307c0" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 46403 } } peers { permanent_uuid: "8cdeb04a5f7c4122bc0faea6e4b57b85" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 39161 } }
I20260925 22:36:21.504968 12246 leader_election.cc:288] T 821633a5289948d1accbd91298ff0c0f P 100e75ff3c2a447491843e27e50516eb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 95fe301fe790464995b14857f26307c0 (127.6.246.194:46403), 8cdeb04a5f7c4122bc0faea6e4b57b85 (127.6.246.195:39161)
I20260925 22:36:21.507187 12236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:21.505259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.193:46051 (local address 127.6.246.194:46403)
0925 22:36:21.505385 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:21.505390 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:21.505652 (+ 262us) server_negotiation.cc:407] Connection header received
0925 22:36:21.505823 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:21.505826 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:21.505866 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:21.505965 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:21.506770 (+ 805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:21.506774 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:21.506777 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:36:21.506790 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:21.506831 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:36:21.506843 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:21.506952 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:21.507019 (+ 67us) server_negotiation.cc:299] Negotiation successful
0925 22:36:21.507059 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 22:36:21.507187 12237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:21.505349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.193:59233 (local address 127.6.246.195:39161)
0925 22:36:21.505473 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:21.505476 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:21.505695 (+ 219us) server_negotiation.cc:407] Connection header received
0925 22:36:21.505823 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:21.505826 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:21.505866 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:21.505957 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:21.506770 (+ 813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:21.506774 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:21.506777 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:36:21.506790 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:21.506813 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:36:21.506824 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:21.506923 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:21.507019 (+ 96us) server_negotiation.cc:299] Negotiation successful
0925 22:36:21.507059 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 22:36:21.507252 12248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:21.505178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.194:46403 (local address 127.6.246.193:46051)
0925 22:36:21.505553 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:21.505568 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:21.505724 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:21.506001 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:21.506005 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:21.506022 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:21.506612 (+ 590us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:36:21.506614 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:21.506621 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:36:21.506637 (+ 16us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:36:21.506934 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:36:21.506937 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:36:21.506939 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 22:36:21.507013 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 22:36:21.507064 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":89,"threads_started":1}
I20260925 22:36:21.507258 12247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:21.505208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.195:39161 (local address 127.6.246.193:59233)
0925 22:36:21.505608 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:21.505621 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:21.505724 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:21.506001 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:21.506005 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:21.506022 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:21.506588 (+ 566us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:36:21.506592 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:21.506606 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:36:21.506637 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:36:21.506916 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:36:21.506921 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:36:21.506926 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 22:36:21.507013 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 22:36:21.507075 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"mutex_wait_us":19,"spinlock_wait_cycles":70912,"thread_start_us":73,"threads_started":1}
I20260925 22:36:21.507998 12044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "821633a5289948d1accbd91298ff0c0f" candidate_uuid: "100e75ff3c2a447491843e27e50516eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95fe301fe790464995b14857f26307c0" is_pre_election: true
I20260925 22:36:21.508010 12175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "821633a5289948d1accbd91298ff0c0f" candidate_uuid: "100e75ff3c2a447491843e27e50516eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cdeb04a5f7c4122bc0faea6e4b57b85" is_pre_election: true
I20260925 22:36:21.508284 12044 raft_consensus.cc:2454] T 821633a5289948d1accbd91298ff0c0f P 95fe301fe790464995b14857f26307c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 100e75ff3c2a447491843e27e50516eb in term 0.
I20260925 22:36:21.508297 12175 raft_consensus.cc:2454] T 821633a5289948d1accbd91298ff0c0f P 8cdeb04a5f7c4122bc0faea6e4b57b85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 100e75ff3c2a447491843e27e50516eb in term 0.
I20260925 22:36:21.508776 11847 leader_election.cc:302] T 821633a5289948d1accbd91298ff0c0f P 100e75ff3c2a447491843e27e50516eb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 100e75ff3c2a447491843e27e50516eb, 95fe301fe790464995b14857f26307c0; no voters:
I20260925 22:36:21.509060 12246 raft_consensus.cc:2781] T 821633a5289948d1accbd91298ff0c0f P 100e75ff3c2a447491843e27e50516eb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:36:21.509148 12246 raft_consensus.cc:482] T 821633a5289948d1accbd91298ff0c0f P 100e75ff3c2a447491843e27e50516eb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:36:21.509198 12246 raft_consensus.cc:3037] T 821633a5289948d1accbd91298ff0c0f P 100e75ff3c2a447491843e27e50516eb [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:36:21.510179 12246 raft_consensus.cc:504] T 821633a5289948d1accbd91298ff0c0f P 100e75ff3c2a447491843e27e50516eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100e75ff3c2a447491843e27e50516eb" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 38659 } } peers { permanent_uuid: "95fe301fe790464995b14857f26307c0" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 46403 } } peers { permanent_uuid: "8cdeb04a5f7c4122bc0faea6e4b57b85" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 39161 } }
I20260925 22:36:21.510488 12246 leader_election.cc:288] T 821633a5289948d1accbd91298ff0c0f P 100e75ff3c2a447491843e27e50516eb [CANDIDATE]: Term 1 election: Requested vote from peers 95fe301fe790464995b14857f26307c0 (127.6.246.194:46403), 8cdeb04a5f7c4122bc0faea6e4b57b85 (127.6.246.195:39161)
I20260925 22:36:21.510784 12044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "821633a5289948d1accbd91298ff0c0f" candidate_uuid: "100e75ff3c2a447491843e27e50516eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95fe301fe790464995b14857f26307c0"
I20260925 22:36:21.510859 12175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "821633a5289948d1accbd91298ff0c0f" candidate_uuid: "100e75ff3c2a447491843e27e50516eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cdeb04a5f7c4122bc0faea6e4b57b85"
I20260925 22:36:21.510941 12044 raft_consensus.cc:3037] T 821633a5289948d1accbd91298ff0c0f P 95fe301fe790464995b14857f26307c0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:36:21.510987 12175 raft_consensus.cc:3037] T 821633a5289948d1accbd91298ff0c0f P 8cdeb04a5f7c4122bc0faea6e4b57b85 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:36:21.511921 12044 raft_consensus.cc:2454] T 821633a5289948d1accbd91298ff0c0f P 95fe301fe790464995b14857f26307c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 100e75ff3c2a447491843e27e50516eb in term 1.
I20260925 22:36:21.512072 12175 raft_consensus.cc:2454] T 821633a5289948d1accbd91298ff0c0f P 8cdeb04a5f7c4122bc0faea6e4b57b85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 100e75ff3c2a447491843e27e50516eb in term 1.
I20260925 22:36:21.512235 11847 leader_election.cc:302] T 821633a5289948d1accbd91298ff0c0f P 100e75ff3c2a447491843e27e50516eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 100e75ff3c2a447491843e27e50516eb, 95fe301fe790464995b14857f26307c0; no voters:
I20260925 22:36:21.512488 12246 raft_consensus.cc:2781] T 821633a5289948d1accbd91298ff0c0f P 100e75ff3c2a447491843e27e50516eb [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:36:21.512677 12246 raft_consensus.cc:686] T 821633a5289948d1accbd91298ff0c0f P 100e75ff3c2a447491843e27e50516eb [term 1 LEADER]: Becoming Leader. State: Replica: 100e75ff3c2a447491843e27e50516eb, State: Running, Role: LEADER
I20260925 22:36:21.513015 12246 consensus_queue.cc:237] T 821633a5289948d1accbd91298ff0c0f P 100e75ff3c2a447491843e27e50516eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100e75ff3c2a447491843e27e50516eb" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 38659 } } peers { permanent_uuid: "95fe301fe790464995b14857f26307c0" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 46403 } } peers { permanent_uuid: "8cdeb04a5f7c4122bc0faea6e4b57b85" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 39161 } }
I20260925 22:36:21.516026 11767 catalog_manager.cc:5495] T 821633a5289948d1accbd91298ff0c0f P 100e75ff3c2a447491843e27e50516eb reported cstate change: term changed from 0 to 1, leader changed from <none> to 100e75ff3c2a447491843e27e50516eb (127.6.246.193). New cstate: current_term: 1 leader_uuid: "100e75ff3c2a447491843e27e50516eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100e75ff3c2a447491843e27e50516eb" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 38659 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95fe301fe790464995b14857f26307c0" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 46403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8cdeb04a5f7c4122bc0faea6e4b57b85" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 39161 } health_report { overall_health: UNKNOWN } } }
I20260925 22:36:21.543519 12233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:21.542141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57542 (local address 127.6.246.193:38659)
0925 22:36:21.542249 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:21.542254 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:21.542268 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:36:21.542362 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:21.542365 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:21.542441 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:21.542551 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:21.543214 (+ 663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:21.543217 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:21.543219 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:36:21.543229 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:21.543245 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:36:21.543253 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:21.543340 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:21.543383 (+ 43us) server_negotiation.cc:299] Negotiation successful
0925 22:36:21.543416 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 22:36:21.549063 12175 raft_consensus.cc:1261] T 821633a5289948d1accbd91298ff0c0f P 8cdeb04a5f7c4122bc0faea6e4b57b85 [term 1 FOLLOWER]: Refusing update from remote peer 100e75ff3c2a447491843e27e50516eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:36:21.549063 12044 raft_consensus.cc:1261] T 821633a5289948d1accbd91298ff0c0f P 95fe301fe790464995b14857f26307c0 [term 1 FOLLOWER]: Refusing update from remote peer 100e75ff3c2a447491843e27e50516eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:36:21.549724 12246 consensus_queue.cc:1035] T 821633a5289948d1accbd91298ff0c0f P 100e75ff3c2a447491843e27e50516eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "95fe301fe790464995b14857f26307c0" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 46403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:36:21.549965 12249 consensus_queue.cc:1035] T 821633a5289948d1accbd91298ff0c0f P 100e75ff3c2a447491843e27e50516eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cdeb04a5f7c4122bc0faea6e4b57b85" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 39161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:36:21.560174 12254 mvcc.cc:205] Tried to move back new op lower bound from 7333379201218301952 to 7333379201079803904. Current Snapshot: MvccSnapshot[applied={T|T < 7333379201218301952}]
I20260925 22:36:21.566588 11771 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45180:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:36:21.566737 11771 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45180:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:36:21.569365 11771 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8b5f50027f574cf9ac678befb1b1ddd5: Sending DeleteTablet for 3 replicas of tablet 821633a5289948d1accbd91298ff0c0f
I20260925 22:36:21.570122 11893 tablet_service.cc:1514] Processing DeleteTablet for tablet 821633a5289948d1accbd91298ff0c0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:36:21 UTC) from {username='slave'} at 127.0.0.1:57528
I20260925 22:36:21.570147 12024 tablet_service.cc:1514] Processing DeleteTablet for tablet 821633a5289948d1accbd91298ff0c0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:36:21 UTC) from {username='slave'} at 127.0.0.1:44096
I20260925 22:36:21.570266 12155 tablet_service.cc:1514] Processing DeleteTablet for tablet 821633a5289948d1accbd91298ff0c0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:36:21 UTC) from {username='slave'} at 127.0.0.1:45506
I20260925 22:36:21.570659 12262 tablet_replica.cc:330] T 821633a5289948d1accbd91298ff0c0f P 100e75ff3c2a447491843e27e50516eb: stopping tablet replica
I20260925 22:36:21.570789 12264 tablet_replica.cc:330] T 821633a5289948d1accbd91298ff0c0f P 8cdeb04a5f7c4122bc0faea6e4b57b85: stopping tablet replica
I20260925 22:36:21.570989 12264 raft_consensus.cc:2229] T 821633a5289948d1accbd91298ff0c0f P 8cdeb04a5f7c4122bc0faea6e4b57b85 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:36:21.570984 12262 raft_consensus.cc:2229] T 821633a5289948d1accbd91298ff0c0f P 100e75ff3c2a447491843e27e50516eb [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:36:21.571267 12264 raft_consensus.cc:2258] T 821633a5289948d1accbd91298ff0c0f P 8cdeb04a5f7c4122bc0faea6e4b57b85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:36:21.571426 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 11831
I20260925 22:36:21.572669 12263 tablet_replica.cc:330] T 821633a5289948d1accbd91298ff0c0f P 95fe301fe790464995b14857f26307c0: stopping tablet replica
I20260925 22:36:21.572835 12264 ts_tablet_manager.cc:1905] T 821633a5289948d1accbd91298ff0c0f P 8cdeb04a5f7c4122bc0faea6e4b57b85: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:36:21.572989 12263 raft_consensus.cc:2229] T 821633a5289948d1accbd91298ff0c0f P 95fe301fe790464995b14857f26307c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:36:21.573266 12263 raft_consensus.cc:2258] T 821633a5289948d1accbd91298ff0c0f P 95fe301fe790464995b14857f26307c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:36:21.575003 12263 ts_tablet_manager.cc:1905] T 821633a5289948d1accbd91298ff0c0f P 95fe301fe790464995b14857f26307c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:36:21.575477 12264 ts_tablet_manager.cc:1918] T 821633a5289948d1accbd91298ff0c0f P 8cdeb04a5f7c4122bc0faea6e4b57b85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:36:21.575541 12264 log.cc:1196] T 821633a5289948d1accbd91298ff0c0f P 8cdeb04a5f7c4122bc0faea6e4b57b85: Deleting WAL directory at /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790375752797504-7131-0/minicluster-data/ts-2/wal/wals/821633a5289948d1accbd91298ff0c0f
I20260925 22:36:21.575752 12264 ts_tablet_manager.cc:1939] T 821633a5289948d1accbd91298ff0c0f P 8cdeb04a5f7c4122bc0faea6e4b57b85: Deleting consensus metadata
I20260925 22:36:21.576916 11757 catalog_manager.cc:4841] TS 8cdeb04a5f7c4122bc0faea6e4b57b85 (127.6.246.195:39161): tablet 821633a5289948d1accbd91298ff0c0f (table test-table [id=eb9308ed3fc0433aaa93b0d3afc73108]) successfully deleted
I20260925 22:36:21.578289 12263 ts_tablet_manager.cc:1918] T 821633a5289948d1accbd91298ff0c0f P 95fe301fe790464995b14857f26307c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:36:21.578392 12263 log.cc:1196] T 821633a5289948d1accbd91298ff0c0f P 95fe301fe790464995b14857f26307c0: Deleting WAL directory at /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790375752797504-7131-0/minicluster-data/ts-1/wal/wals/821633a5289948d1accbd91298ff0c0f
I20260925 22:36:21.578696 12263 ts_tablet_manager.cc:1939] T 821633a5289948d1accbd91298ff0c0f P 95fe301fe790464995b14857f26307c0: Deleting consensus metadata
W20260925 22:36:21.579932 11755 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.246.193:38659 (error 108)
I20260925 22:36:21.580073 11756 catalog_manager.cc:4841] TS 95fe301fe790464995b14857f26307c0 (127.6.246.194:46403): tablet 821633a5289948d1accbd91298ff0c0f (table test-table [id=eb9308ed3fc0433aaa93b0d3afc73108]) successfully deleted
I20260925 22:36:21.580237 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 11962
I20260925 22:36:21.581697 12232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:21.581339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.193:38659 (local address 127.0.0.1:57554)
0925 22:36:21.581511 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:21.581595 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.193:38659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260925 22:36:21.581919 11755 catalog_manager.cc:4584] TS 100e75ff3c2a447491843e27e50516eb (127.6.246.193:38659): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 821633a5289948d1accbd91298ff0c0f: Network error: Client connection negotiation failed: client connection to 127.6.246.193:38659: connect: Connection refused (error 111)
I20260925 22:36:21.586273 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 12093
I20260925 22:36:21.591687 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 11740
2026-09-25T22:36:21Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1066 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260925 22:36:21.612957 7131 mini_oidc.cc:152] Starting JWKS server
I20260925 22:36:21.614481 7131 webserver.cc:460] Webserver started at https://127.0.0.1:40593/ using document root <none> and password file <none>
I20260925 22:36:21.614610 7131 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 22:36:21.614827 7131 webserver.cc:460] Webserver started at http://127.0.0.1:39081/ using document root <none> and password file <none>
2026-09-25T22:36:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:36:21Z Disabled control of system clock
I20260925 22:36:21.623966 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790375752797504-7131-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790375752797504-7131-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790375752797504-7131-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:35729
--webserver_interface=127.6.246.254
--webserver_port=0
--builtin_ntp_servers=127.6.246.212:35785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.246.254:35729
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40593/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790375752797504-7131-0/testchainca.cert with env {}
W20260925 22:36:21.732941 12271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:36:21.733290 12271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:36:21.733351 12271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:36:21.738364 12271 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:36:21.738443 12271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:36:21.738461 12271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:36:21.738479 12271 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:36:21.738494 12271 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 22:36:21.738510 12271 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 22:36:21.738523 12271 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40593/jwks.json
I20260925 22:36:21.743913 12271 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:35785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790375752797504-7131-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790375752797504-7131-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.246.254:35729
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:35729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40593/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.246.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790375752797504-7131-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12271
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790375752797504-7131-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:36:21.744957 12271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:36:21.745903 12271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:36:21.754108 12277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:21.754103 12279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:21.754195 12276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:21.754904 12271 server_base.cc:1034] running on GCE node
I20260925 22:36:21.755452 12271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:36:21.756594 12271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:36:21.757788 12271 hybrid_clock.cc:648] HybridClock initialized: now 1790375781757778 us; error 39 us; skew 500 ppm
I20260925 22:36:21.759814 12271 webserver.cc:460] Webserver started at http://127.6.246.254:39321/ using document root <none> and password file <none>
I20260925 22:36:21.760427 12271 fs_manager.cc:359] Metadata directory not provided
I20260925 22:36:21.760571 12271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:36:21.760799 12271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:36:21.762773 12271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790375752797504-7131-0/minicluster-data/master-0/data/instance:
uuid: "5a18b01a75534109985003fca22b2ae8"
format_stamp: "Formatted at 2026-09-25 22:36:21 on dist-test-slave-wjdr"
server_key: "5485670ecb492bbccaeb7012508a5bdb"
server_key_iv: "95cb5695ba616b318d2fda3d315dee03"
server_key_version: "encryptionkey@0"
I20260925 22:36:21.763335 12271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790375752797504-7131-0/minicluster-data/master-0/wal/instance:
uuid: "5a18b01a75534109985003fca22b2ae8"
format_stamp: "Formatted at 2026-09-25 22:36:21 on dist-test-slave-wjdr"
server_key: "5485670ecb492bbccaeb7012508a5bdb"
server_key_iv: "95cb5695ba616b318d2fda3d315dee03"
server_key_version: "encryptionkey@0"
I20260925 22:36:21.767026 12271 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 22:36:21.769632 12285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:21.770777 12271 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:36:21.770918 12271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790375752797504-7131-0/minicluster-data/master-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790375752797504-7131-0/minicluster-data/master-0/wal
uuid: "5a18b01a75534109985003fca22b2ae8"
format_stamp: "Formatted at 2026-09-25 22:36:21 on dist-test-slave-wjdr"
server_key: "5485670ecb492bbccaeb7012508a5bdb"
server_key_iv: "95cb5695ba616b318d2fda3d315dee03"
server_key_version: "encryptionkey@0"
I20260925 22:36:21.771029 12271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790375752797504-7131-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790375752797504-7131-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790375752797504-7131-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:36:21.789180 12271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:36:21.789887 12271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:36:21.790079 12271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:36:21.796907 12271 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.254:35729
I20260925 22:36:21.796914 12337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.254:35729 every 8 connection(s)
I20260925 22:36:21.797861 12271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
I20260925 22:36:21.799556 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 12271
I20260925 22:36:21.799665 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790375752797504-7131-0/minicluster-data/master-0/wal/instance
I20260925 22:36:21.799993 7131 external_mini_cluster.cc:1428] Setting key 7eaf4d24e1630196e0c15a387aa071f1
I20260925 22:36:21.801504 12338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:36:21.806828 12338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a18b01a75534109985003fca22b2ae8: Bootstrap starting.
I20260925 22:36:21.809038 12341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:21.801411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43442 (local address 127.6.246.254:35729)
0925 22:36:21.802070 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:21.802084 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:21.802119 (+ 35us) server_negotiation.cc:407] Connection header received
0925 22:36:21.802748 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:21.802763 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:21.803135 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:21.803461 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:21.803871 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:21.804879 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:21.805763 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:21.806228 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:21.806400 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:21.807287 (+ 887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:21.807291 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:21.807303 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:36:21.807671 (+ 368us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:21.807722 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:36:21.807741 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:21.807816 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:21.808004 (+ 188us) server_negotiation.cc:299] Negotiation successful
0925 22:36:21.808274 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":141,"threads_started":1}
I20260925 22:36:21.809298 12338 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a18b01a75534109985003fca22b2ae8: Neither blocks nor log segments found. Creating new log.
I20260925 22:36:21.810010 12338 log.cc:824] T 00000000000000000000000000000000 P 5a18b01a75534109985003fca22b2ae8: Log is configured to *not* fsync() on all Append() calls
I20260925 22:36:21.811707 7131 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:36:21.811976 12338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a18b01a75534109985003fca22b2ae8: No bootstrap required, opened a new log
I20260925 22:36:21.815529 12338 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a18b01a75534109985003fca22b2ae8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a18b01a75534109985003fca22b2ae8" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 35729 } }
I20260925 22:36:21.815750 12338 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a18b01a75534109985003fca22b2ae8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:36:21.815786 12338 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a18b01a75534109985003fca22b2ae8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a18b01a75534109985003fca22b2ae8, State: Initialized, Role: FOLLOWER
I20260925 22:36:21.816268 12338 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a18b01a75534109985003fca22b2ae8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a18b01a75534109985003fca22b2ae8" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 35729 } }
I20260925 22:36:21.816402 12338 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a18b01a75534109985003fca22b2ae8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:36:21.816501 12338 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a18b01a75534109985003fca22b2ae8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:36:21.816612 12338 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a18b01a75534109985003fca22b2ae8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:36:21.817553 12338 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a18b01a75534109985003fca22b2ae8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a18b01a75534109985003fca22b2ae8" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 35729 } }
I20260925 22:36:21.817914 12338 leader_election.cc:302] T 00000000000000000000000000000000 P 5a18b01a75534109985003fca22b2ae8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a18b01a75534109985003fca22b2ae8; no voters:
I20260925 22:36:21.818229 12338 leader_election.cc:288] T 00000000000000000000000000000000 P 5a18b01a75534109985003fca22b2ae8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:36:21.818403 12343 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a18b01a75534109985003fca22b2ae8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:36:21.818660 12343 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a18b01a75534109985003fca22b2ae8 [term 1 LEADER]: Becoming Leader. State: Replica: 5a18b01a75534109985003fca22b2ae8, State: Running, Role: LEADER
I20260925 22:36:21.819063 12343 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a18b01a75534109985003fca22b2ae8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a18b01a75534109985003fca22b2ae8" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 35729 } }
I20260925 22:36:21.819615 12338 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a18b01a75534109985003fca22b2ae8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:36:21.820979 12345 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a18b01a75534109985003fca22b2ae8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a18b01a75534109985003fca22b2ae8. Latest consensus state: current_term: 1 leader_uuid: "5a18b01a75534109985003fca22b2ae8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a18b01a75534109985003fca22b2ae8" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 35729 } } }
I20260925 22:36:21.820979 12344 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a18b01a75534109985003fca22b2ae8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a18b01a75534109985003fca22b2ae8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a18b01a75534109985003fca22b2ae8" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 35729 } } }
I20260925 22:36:21.821138 12345 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a18b01a75534109985003fca22b2ae8 [sys.catalog]: This master's current role is: LEADER
I20260925 22:36:21.821141 12344 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a18b01a75534109985003fca22b2ae8 [sys.catalog]: This master's current role is: LEADER
I20260925 22:36:21.821616 12353 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:36:21.824327 12353 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:36:21.829989 12353 catalog_manager.cc:1347] Generated new cluster ID: 44f8141e0c7d43ac8a1da263ba5a5720
I20260925 22:36:21.830052 12353 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:36:21.857529 12353 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:36:21.858485 12353 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:36:21.863462 12353 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a18b01a75534109985003fca22b2ae8: Generated new TSK 0
I20260925 22:36:21.864127 12353 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 22:36:21.888500 12368 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260925 22:36:21.888628 12368 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 22:36:21.889119 12341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:21.872491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43452 (local address 127.6.246.254:35729)
0925 22:36:21.872614 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:21.872619 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:21.872815 (+ 196us) server_negotiation.cc:407] Connection header received
0925 22:36:21.873035 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:21.873037 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:21.873084 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:21.873171 (+ 87us) server_negotiation.cc:226] Negotiated authn=JWT
0925 22:36:21.873440 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:21.873980 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:21.875207 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:21.875430 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:21.875509 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:21.875857 (+ 348us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 22:36:21.888782 (+ 12925us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40593/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 22:36:21.889012 (+ 230us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43452: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40593/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":36}
W20260925 22:36:21.889216 12341 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43452: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40593/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260925 22:36:21.890559 12367 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:36:21.872335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:35729 (local address 127.0.0.1:43452)
0925 22:36:21.872718 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:21.872734 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:21.872875 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:21.873204 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:21.873207 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:21.873215 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0925 22:36:21.873336 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:21.873342 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:21.874099 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:21.874102 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:21.875095 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:21.875105 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:21.875549 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:21.875553 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:21.875648 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:21.875699 (+ 51us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 22:36:21.889022 (+ 13323us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40593/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 22:36:21.889261 (+ 239us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.246.254:35729: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40593/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":129,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.6.246.254:35729: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40593/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260925 22:36:21.891582 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 12271
2026-09-25T22:36:21Z chronyd exiting
I20260925 22:36:21.928951 7131 test_util.cc:175] -----------------------------------------------
I20260925 22:36:21.929040 7131 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790375752797504-7131-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (317 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260925 22:36:21.930078 7131 mini_oidc.cc:152] Starting JWKS server
I20260925 22:36:21.930932 7131 webserver.cc:460] Webserver started at https://127.0.0.1:35213/ using document root <none> and password file <none>
I20260925 22:36:21.931035 7131 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 22:36:21.931334 7131 webserver.cc:460] Webserver started at http://127.0.0.1:33401/ using document root <none> and password file <none>
2026-09-25T22:36:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:36:21Z Disabled control of system clock
I20260925 22:36:21.939973 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790375752797504-7131-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790375752797504-7131-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790375752797504-7131-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:34653
--webserver_interface=127.6.246.254
--webserver_port=0
--builtin_ntp_servers=127.6.246.212:36663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.246.254:34653
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35213/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790375752797504-7131-0/testchainca.cert with env {}
W20260925 22:36:22.051542 12374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:36:22.051880 12374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:36:22.051954 12374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:36:22.056157 12374 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:36:22.056250 12374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:36:22.056278 12374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:36:22.056318 12374 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:36:22.056354 12374 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 22:36:22.056389 12374 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 22:36:22.056420 12374 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35213/jwks.json
I20260925 22:36:22.061947 12374 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:36663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790375752797504-7131-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790375752797504-7131-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.246.254:34653
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:34653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35213/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.246.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790375752797504-7131-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790375752797504-7131-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:36:22.063016 12374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:36:22.063997 12374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:36:22.072525 12379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:22.072573 12380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:22.072546 12382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:22.072717 12374 server_base.cc:1034] running on GCE node
I20260925 22:36:22.073469 12374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:36:22.074544 12374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:36:22.075706 12374 hybrid_clock.cc:648] HybridClock initialized: now 1790375782075688 us; error 32 us; skew 500 ppm
I20260925 22:36:22.077476 12374 webserver.cc:460] Webserver started at http://127.6.246.254:35851/ using document root <none> and password file <none>
I20260925 22:36:22.077993 12374 fs_manager.cc:359] Metadata directory not provided
I20260925 22:36:22.078048 12374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:36:22.078204 12374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:36:22.080153 12374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790375752797504-7131-0/minicluster-data/master-0/data/instance:
uuid: "cca2ff1b8ede45f397d333408e6eb725"
format_stamp: "Formatted at 2026-09-25 22:36:22 on dist-test-slave-wjdr"
server_key: "6a337635eded604ed3f2e1fb8075ca67"
server_key_iv: "3c633f464b10d604f390f3b3bb7a20b6"
server_key_version: "encryptionkey@0"
I20260925 22:36:22.080729 12374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790375752797504-7131-0/minicluster-data/master-0/wal/instance:
uuid: "cca2ff1b8ede45f397d333408e6eb725"
format_stamp: "Formatted at 2026-09-25 22:36:22 on dist-test-slave-wjdr"
server_key: "6a337635eded604ed3f2e1fb8075ca67"
server_key_iv: "3c633f464b10d604f390f3b3bb7a20b6"
server_key_version: "encryptionkey@0"
I20260925 22:36:22.084205 12374 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260925 22:36:22.086719 12388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:22.087884 12374 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:36:22.088020 12374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790375752797504-7131-0/minicluster-data/master-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790375752797504-7131-0/minicluster-data/master-0/wal
uuid: "cca2ff1b8ede45f397d333408e6eb725"
format_stamp: "Formatted at 2026-09-25 22:36:22 on dist-test-slave-wjdr"
server_key: "6a337635eded604ed3f2e1fb8075ca67"
server_key_iv: "3c633f464b10d604f390f3b3bb7a20b6"
server_key_version: "encryptionkey@0"
I20260925 22:36:22.088135 12374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790375752797504-7131-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790375752797504-7131-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790375752797504-7131-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:36:22.105695 12374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:36:22.106395 12374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:36:22.106595 12374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:36:22.113443 12374 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.254:34653
I20260925 22:36:22.113451 12440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.254:34653 every 8 connection(s)
I20260925 22:36:22.114377 12374 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
I20260925 22:36:22.115466 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 12374
I20260925 22:36:22.115614 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790375752797504-7131-0/minicluster-data/master-0/wal/instance
I20260925 22:36:22.115904 7131 external_mini_cluster.cc:1428] Setting key 40195c1fc7c74a64f9d8cbd1aa5fe04d
I20260925 22:36:22.118505 12441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:36:22.124794 12441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cca2ff1b8ede45f397d333408e6eb725: Bootstrap starting.
I20260925 22:36:22.125350 12444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:22.117227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46580 (local address 127.6.246.254:34653)
0925 22:36:22.117843 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:22.117858 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:22.117894 (+ 36us) server_negotiation.cc:407] Connection header received
0925 22:36:22.118528 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:22.118584 (+ 56us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:22.118993 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:22.119266 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:22.119609 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:22.120656 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:22.121897 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:22.122358 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:22.122484 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:22.123434 (+ 950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:22.123439 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:22.123450 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:36:22.123821 (+ 371us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:22.123909 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:36:22.123931 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:22.124013 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:22.124206 (+ 193us) server_negotiation.cc:299] Negotiation successful
0925 22:36:22.124568 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":110,"threads_started":1}
I20260925 22:36:22.127197 12441 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cca2ff1b8ede45f397d333408e6eb725: Neither blocks nor log segments found. Creating new log.
I20260925 22:36:22.128096 12441 log.cc:824] T 00000000000000000000000000000000 P cca2ff1b8ede45f397d333408e6eb725: Log is configured to *not* fsync() on all Append() calls
I20260925 22:36:22.128181 7131 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:36:22.130250 12441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cca2ff1b8ede45f397d333408e6eb725: No bootstrap required, opened a new log
I20260925 22:36:22.133807 12441 raft_consensus.cc:348] T 00000000000000000000000000000000 P cca2ff1b8ede45f397d333408e6eb725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cca2ff1b8ede45f397d333408e6eb725" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 34653 } }
I20260925 22:36:22.134037 12441 raft_consensus.cc:374] T 00000000000000000000000000000000 P cca2ff1b8ede45f397d333408e6eb725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:36:22.134119 12441 raft_consensus.cc:729] T 00000000000000000000000000000000 P cca2ff1b8ede45f397d333408e6eb725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca2ff1b8ede45f397d333408e6eb725, State: Initialized, Role: FOLLOWER
I20260925 22:36:22.134558 12441 consensus_queue.cc:260] T 00000000000000000000000000000000 P cca2ff1b8ede45f397d333408e6eb725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cca2ff1b8ede45f397d333408e6eb725" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 34653 } }
I20260925 22:36:22.134722 12441 raft_consensus.cc:388] T 00000000000000000000000000000000 P cca2ff1b8ede45f397d333408e6eb725 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:36:22.134811 12441 raft_consensus.cc:482] T 00000000000000000000000000000000 P cca2ff1b8ede45f397d333408e6eb725 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:36:22.134912 12441 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cca2ff1b8ede45f397d333408e6eb725 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:36:22.135773 12441 raft_consensus.cc:504] T 00000000000000000000000000000000 P cca2ff1b8ede45f397d333408e6eb725 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cca2ff1b8ede45f397d333408e6eb725" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 34653 } }
I20260925 22:36:22.136088 12441 leader_election.cc:302] T 00000000000000000000000000000000 P cca2ff1b8ede45f397d333408e6eb725 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cca2ff1b8ede45f397d333408e6eb725; no voters:
I20260925 22:36:22.136387 12441 leader_election.cc:288] T 00000000000000000000000000000000 P cca2ff1b8ede45f397d333408e6eb725 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:36:22.136533 12446 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cca2ff1b8ede45f397d333408e6eb725 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:36:22.136767 12446 raft_consensus.cc:686] T 00000000000000000000000000000000 P cca2ff1b8ede45f397d333408e6eb725 [term 1 LEADER]: Becoming Leader. State: Replica: cca2ff1b8ede45f397d333408e6eb725, State: Running, Role: LEADER
I20260925 22:36:22.137068 12446 consensus_queue.cc:237] T 00000000000000000000000000000000 P cca2ff1b8ede45f397d333408e6eb725 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cca2ff1b8ede45f397d333408e6eb725" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 34653 } }
I20260925 22:36:22.137574 12441 sys_catalog.cc:564] T 00000000000000000000000000000000 P cca2ff1b8ede45f397d333408e6eb725 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:36:22.138628 12448 sys_catalog.cc:455] T 00000000000000000000000000000000 P cca2ff1b8ede45f397d333408e6eb725 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cca2ff1b8ede45f397d333408e6eb725. Latest consensus state: current_term: 1 leader_uuid: "cca2ff1b8ede45f397d333408e6eb725" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cca2ff1b8ede45f397d333408e6eb725" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 34653 } } }
I20260925 22:36:22.138752 12448 sys_catalog.cc:458] T 00000000000000000000000000000000 P cca2ff1b8ede45f397d333408e6eb725 [sys.catalog]: This master's current role is: LEADER
I20260925 22:36:22.139057 12447 sys_catalog.cc:455] T 00000000000000000000000000000000 P cca2ff1b8ede45f397d333408e6eb725 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cca2ff1b8ede45f397d333408e6eb725" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cca2ff1b8ede45f397d333408e6eb725" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 34653 } } }
I20260925 22:36:22.139158 12447 sys_catalog.cc:458] T 00000000000000000000000000000000 P cca2ff1b8ede45f397d333408e6eb725 [sys.catalog]: This master's current role is: LEADER
I20260925 22:36:22.139165 12455 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:36:22.142377 12455 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:36:22.148018 12455 catalog_manager.cc:1347] Generated new cluster ID: f26942128df64ff08e3b404d6c9c1fa1
I20260925 22:36:22.148084 12455 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:36:22.165882 12455 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:36:22.166793 12455 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:36:22.176076 12455 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cca2ff1b8ede45f397d333408e6eb725: Generated new TSK 0
I20260925 22:36:22.176791 12455 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 22:36:22.214953 12471 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260925 22:36:22.215055 12471 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 22:36:22.215678 12444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:22.192356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46590 (local address 127.6.246.254:34653)
0925 22:36:22.192534 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:22.192539 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:22.192718 (+ 179us) server_negotiation.cc:407] Connection header received
0925 22:36:22.192896 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:22.192898 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:22.192947 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:22.193036 (+ 89us) server_negotiation.cc:226] Negotiated authn=JWT
0925 22:36:22.193286 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:22.193804 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:22.195087 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:22.195316 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:22.195405 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:22.195731 (+ 326us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 22:36:22.215318 (+ 19587us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35213/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 22:36:22.215558 (+ 240us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46590: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35213/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":85}
W20260925 22:36:22.215762 12444 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46590: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35213/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260925 22:36:22.215876 12470 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:36:22.192193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:34653 (local address 127.0.0.1:46590)
0925 22:36:22.192613 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:22.192629 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:22.192745 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:22.193059 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:22.193062 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:22.193070 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0925 22:36:22.193185 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:22.193191 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:22.193943 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:22.193946 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:22.194938 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:22.194950 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:22.195438 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:22.195442 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:22.195532 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:22.195559 (+ 27us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 22:36:22.215521 (+ 19962us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35213/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 22:36:22.215697 (+ 176us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.246.254:34653: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35213/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":242,"spinlock_wait_cycles":25856,"thread_start_us":120,"threads_started":1}
I20260925 22:36:22.216675 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 12374
2026-09-25T22:36:22Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (325 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260925 22:36:22.255854 7131 mini_oidc.cc:152] Starting JWKS server
I20260925 22:36:22.256783 7131 webserver.cc:460] Webserver started at https://127.0.0.1:39837/ using document root <none> and password file <none>
I20260925 22:36:22.256908 7131 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 22:36:22.257185 7131 webserver.cc:460] Webserver started at http://127.0.0.1:45297/ using document root <none> and password file <none>
2026-09-25T22:36:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:36:22Z Disabled control of system clock
I20260925 22:36:22.266062 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790375752797504-7131-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790375752797504-7131-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790375752797504-7131-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:45377
--webserver_interface=127.6.246.254
--webserver_port=0
--builtin_ntp_servers=127.6.246.212:39751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.246.254:45377
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39837/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 22:36:22.375382 12477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:36:22.375638 12477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:36:22.375677 12477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:36:22.379593 12477 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:36:22.379652 12477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:36:22.379667 12477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:36:22.379683 12477 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:36:22.379698 12477 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 22:36:22.379714 12477 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 22:36:22.379729 12477 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39837/jwks.json
I20260925 22:36:22.384474 12477 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:39751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790375752797504-7131-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790375752797504-7131-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.246.254:45377
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:45377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39837/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.246.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790375752797504-7131-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:36:22.385658 12477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:36:22.387068 12477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:36:22.395874 12483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:22.395836 12485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:22.395949 12482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:22.396054 12477 server_base.cc:1034] running on GCE node
I20260925 22:36:22.396710 12477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:36:22.397750 12477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:36:22.398927 12477 hybrid_clock.cc:648] HybridClock initialized: now 1790375782398897 us; error 41 us; skew 500 ppm
I20260925 22:36:22.400947 12477 webserver.cc:460] Webserver started at http://127.6.246.254:45573/ using document root <none> and password file <none>
I20260925 22:36:22.401510 12477 fs_manager.cc:359] Metadata directory not provided
I20260925 22:36:22.401571 12477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:36:22.401787 12477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:36:22.403812 12477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790375752797504-7131-0/minicluster-data/master-0/data/instance:
uuid: "8a5181db0b7d4d80bf1428e419999682"
format_stamp: "Formatted at 2026-09-25 22:36:22 on dist-test-slave-wjdr"
server_key: "b2621923a7cc20d5b7e94ab25dc91a6b"
server_key_iv: "e701a7a3464f172885914eb4f2c1bdb4"
server_key_version: "encryptionkey@0"
I20260925 22:36:22.404376 12477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790375752797504-7131-0/minicluster-data/master-0/wal/instance:
uuid: "8a5181db0b7d4d80bf1428e419999682"
format_stamp: "Formatted at 2026-09-25 22:36:22 on dist-test-slave-wjdr"
server_key: "b2621923a7cc20d5b7e94ab25dc91a6b"
server_key_iv: "e701a7a3464f172885914eb4f2c1bdb4"
server_key_version: "encryptionkey@0"
I20260925 22:36:22.407948 12477 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:36:22.410327 12491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:22.411588 12477 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:36:22.411723 12477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790375752797504-7131-0/minicluster-data/master-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790375752797504-7131-0/minicluster-data/master-0/wal
uuid: "8a5181db0b7d4d80bf1428e419999682"
format_stamp: "Formatted at 2026-09-25 22:36:22 on dist-test-slave-wjdr"
server_key: "b2621923a7cc20d5b7e94ab25dc91a6b"
server_key_iv: "e701a7a3464f172885914eb4f2c1bdb4"
server_key_version: "encryptionkey@0"
I20260925 22:36:22.411863 12477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790375752797504-7131-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790375752797504-7131-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790375752797504-7131-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:36:22.432757 12477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:36:22.433475 12477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:36:22.433676 12477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:36:22.440886 12477 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.254:45377
I20260925 22:36:22.440912 12543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.254:45377 every 8 connection(s)
I20260925 22:36:22.441861 12477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
I20260925 22:36:22.444892 12544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:36:22.450133 12544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a5181db0b7d4d80bf1428e419999682: Bootstrap starting.
I20260925 22:36:22.451790 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 12477
I20260925 22:36:22.451965 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790375752797504-7131-0/minicluster-data/master-0/wal/instance
I20260925 22:36:22.452246 7131 external_mini_cluster.cc:1428] Setting key 984833098de60aff9dc3609877e33041
I20260925 22:36:22.452656 12544 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a5181db0b7d4d80bf1428e419999682: Neither blocks nor log segments found. Creating new log.
I20260925 22:36:22.453500 12544 log.cc:824] T 00000000000000000000000000000000 P 8a5181db0b7d4d80bf1428e419999682: Log is configured to *not* fsync() on all Append() calls
I20260925 22:36:22.455449 12544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a5181db0b7d4d80bf1428e419999682: No bootstrap required, opened a new log
I20260925 22:36:22.459609 12544 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a5181db0b7d4d80bf1428e419999682 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a5181db0b7d4d80bf1428e419999682" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 45377 } }
I20260925 22:36:22.459851 12544 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a5181db0b7d4d80bf1428e419999682 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:36:22.459932 12544 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a5181db0b7d4d80bf1428e419999682 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a5181db0b7d4d80bf1428e419999682, State: Initialized, Role: FOLLOWER
I20260925 22:36:22.460352 12544 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a5181db0b7d4d80bf1428e419999682 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a5181db0b7d4d80bf1428e419999682" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 45377 } }
I20260925 22:36:22.460511 12544 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a5181db0b7d4d80bf1428e419999682 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:36:22.460601 12544 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a5181db0b7d4d80bf1428e419999682 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:36:22.460724 12544 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a5181db0b7d4d80bf1428e419999682 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:36:22.461598 12544 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a5181db0b7d4d80bf1428e419999682 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a5181db0b7d4d80bf1428e419999682" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 45377 } }
I20260925 22:36:22.461711 12547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:22.453384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60668 (local address 127.6.246.254:45377)
0925 22:36:22.454128 (+ 744us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:22.454147 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:22.454190 (+ 43us) server_negotiation.cc:407] Connection header received
0925 22:36:22.454864 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:22.454880 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:22.455199 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:22.455474 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:22.455889 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:22.456922 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:22.458238 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:22.458744 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:22.458883 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:22.459741 (+ 858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:22.459745 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:22.459794 (+ 49us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:36:22.460355 (+ 561us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:22.460412 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:36:22.460432 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:22.460575 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:22.460726 (+ 151us) server_negotiation.cc:299] Negotiation successful
0925 22:36:22.461018 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":129,"threads_started":1}
I20260925 22:36:22.461997 12544 leader_election.cc:302] T 00000000000000000000000000000000 P 8a5181db0b7d4d80bf1428e419999682 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a5181db0b7d4d80bf1428e419999682; no voters:
I20260925 22:36:22.462314 12544 leader_election.cc:288] T 00000000000000000000000000000000 P 8a5181db0b7d4d80bf1428e419999682 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:36:22.462507 12549 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a5181db0b7d4d80bf1428e419999682 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:36:22.462740 12549 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a5181db0b7d4d80bf1428e419999682 [term 1 LEADER]: Becoming Leader. State: Replica: 8a5181db0b7d4d80bf1428e419999682, State: Running, Role: LEADER
I20260925 22:36:22.463045 12549 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a5181db0b7d4d80bf1428e419999682 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a5181db0b7d4d80bf1428e419999682" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 45377 } }
I20260925 22:36:22.463508 12544 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a5181db0b7d4d80bf1428e419999682 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:36:22.464896 7131 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:36:22.464854 12550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a5181db0b7d4d80bf1428e419999682 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a5181db0b7d4d80bf1428e419999682" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a5181db0b7d4d80bf1428e419999682" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 45377 } } }
I20260925 22:36:22.464975 12550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a5181db0b7d4d80bf1428e419999682 [sys.catalog]: This master's current role is: LEADER
I20260925 22:36:22.465389 12559 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:36:22.465849 12551 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a5181db0b7d4d80bf1428e419999682 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a5181db0b7d4d80bf1428e419999682. Latest consensus state: current_term: 1 leader_uuid: "8a5181db0b7d4d80bf1428e419999682" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a5181db0b7d4d80bf1428e419999682" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 45377 } } }
I20260925 22:36:22.466102 12551 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a5181db0b7d4d80bf1428e419999682 [sys.catalog]: This master's current role is: LEADER
I20260925 22:36:22.468268 12559 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:36:22.473764 12559 catalog_manager.cc:1347] Generated new cluster ID: 1d7af6c3179b4caf8991fd850493261e
I20260925 22:36:22.473904 12559 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:36:22.492488 12559 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:36:22.493381 12559 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:36:22.500639 12559 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a5181db0b7d4d80bf1428e419999682: Generated new TSK 0
I20260925 22:36:22.501284 12559 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:36:22.517432 12547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:22.512184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60678 (local address 127.6.246.254:45377)
0925 22:36:22.512312 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:22.512316 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:22.512518 (+ 202us) server_negotiation.cc:407] Connection header received
0925 22:36:22.512704 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:22.512707 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:22.512766 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:22.512887 (+ 121us) server_negotiation.cc:226] Negotiated authn=JWT
0925 22:36:22.513083 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:22.513923 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:22.514759 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:22.515118 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:22.515219 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:22.515586 (+ 367us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 22:36:22.517078 (+ 1492us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39837/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 22:36:22.517301 (+ 223us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60678: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39837/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":48}
W20260925 22:36:22.517514 12547 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60678: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39837/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260925 22:36:22.519084 12573 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:36:22.512112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:45377 (local address 127.0.0.1:60678)
0925 22:36:22.512409 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:22.512421 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:22.512579 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:22.512888 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:22.512890 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:22.512896 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0925 22:36:22.512987 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:22.512991 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:22.514037 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:22.514040 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:22.514652 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:22.514659 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:22.515228 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:22.515230 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:22.515343 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:22.515408 (+ 65us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 22:36:22.517239 (+ 1831us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39837/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 22:36:22.518917 (+ 1678us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.246.254:45377: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39837/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":1494,"thread_start_us":99,"threads_started":1}
I20260925 22:36:22.519672 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 12477
E20260925 22:36:22.530256 12574 webserver.cc:533] Webserver: sslize failed:
E20260925 22:36:22.530350 12574 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-25T22:36:22Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (304 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-25T22:36:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:36:22Z Disabled control of system clock
I20260925 22:36:22.569078 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790375752797504-7131-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790375752797504-7131-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790375752797504-7131-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:36551
--webserver_interface=127.6.246.254
--webserver_port=0
--builtin_ntp_servers=127.6.246.212:44459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.246.254:36551
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790375752797504-7131-0/testchainca.cert with env {}
W20260925 22:36:22.677150 12578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:36:22.677412 12578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:36:22.677451 12578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:36:22.681402 12578 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:36:22.681453 12578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:36:22.681469 12578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:36:22.681485 12578 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:36:22.681500 12578 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 22:36:22.681559 12578 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 22:36:22.681578 12578 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260925 22:36:22.686946 12578 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:44459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790375752797504-7131-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790375752797504-7131-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.246.254:36551
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:36551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.246.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790375752797504-7131-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790375752797504-7131-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:36:22.687933 12578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:36:22.688915 12578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:36:22.696954 12584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:22.697049 12586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:22.697036 12578 server_base.cc:1034] running on GCE node
W20260925 22:36:22.696998 12583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:22.697742 12578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:36:22.698794 12578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:36:22.699986 12578 hybrid_clock.cc:648] HybridClock initialized: now 1790375782699973 us; error 34 us; skew 500 ppm
I20260925 22:36:22.701745 12578 webserver.cc:460] Webserver started at http://127.6.246.254:37371/ using document root <none> and password file <none>
I20260925 22:36:22.702304 12578 fs_manager.cc:359] Metadata directory not provided
I20260925 22:36:22.702405 12578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:36:22.702626 12578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:36:22.704602 12578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790375752797504-7131-0/minicluster-data/master-0/data/instance:
uuid: "76c34f8036954297be2d65e2cd1a21d9"
format_stamp: "Formatted at 2026-09-25 22:36:22 on dist-test-slave-wjdr"
server_key: "a4bbe5c73c2058eec4ceb67ec6d3493e"
server_key_iv: "1769c4282d932da11ef5bc16557f215b"
server_key_version: "encryptionkey@0"
I20260925 22:36:22.705161 12578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790375752797504-7131-0/minicluster-data/master-0/wal/instance:
uuid: "76c34f8036954297be2d65e2cd1a21d9"
format_stamp: "Formatted at 2026-09-25 22:36:22 on dist-test-slave-wjdr"
server_key: "a4bbe5c73c2058eec4ceb67ec6d3493e"
server_key_iv: "1769c4282d932da11ef5bc16557f215b"
server_key_version: "encryptionkey@0"
I20260925 22:36:22.708645 12578 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 22:36:22.710871 12592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:22.711982 12578 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:36:22.712117 12578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790375752797504-7131-0/minicluster-data/master-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790375752797504-7131-0/minicluster-data/master-0/wal
uuid: "76c34f8036954297be2d65e2cd1a21d9"
format_stamp: "Formatted at 2026-09-25 22:36:22 on dist-test-slave-wjdr"
server_key: "a4bbe5c73c2058eec4ceb67ec6d3493e"
server_key_iv: "1769c4282d932da11ef5bc16557f215b"
server_key_version: "encryptionkey@0"
I20260925 22:36:22.712227 12578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790375752797504-7131-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790375752797504-7131-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790375752797504-7131-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:36:22.735793 12578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:36:22.736740 12578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:36:22.736946 12578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:36:22.744599 12578 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.254:36551
I20260925 22:36:22.744621 12644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.254:36551 every 8 connection(s)
I20260925 22:36:22.745508 12578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
I20260925 22:36:22.748561 12645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:36:22.753899 12645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76c34f8036954297be2d65e2cd1a21d9: Bootstrap starting.
I20260925 22:36:22.754329 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 12578
I20260925 22:36:22.754449 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790375752797504-7131-0/minicluster-data/master-0/wal/instance
I20260925 22:36:22.754739 7131 external_mini_cluster.cc:1428] Setting key 8e91cfed160a72c4eee49c54ecf96314
I20260925 22:36:22.756630 12645 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76c34f8036954297be2d65e2cd1a21d9: Neither blocks nor log segments found. Creating new log.
I20260925 22:36:22.757357 12645 log.cc:824] T 00000000000000000000000000000000 P 76c34f8036954297be2d65e2cd1a21d9: Log is configured to *not* fsync() on all Append() calls
I20260925 22:36:22.759164 12645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76c34f8036954297be2d65e2cd1a21d9: No bootstrap required, opened a new log
I20260925 22:36:22.762764 12645 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76c34f8036954297be2d65e2cd1a21d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76c34f8036954297be2d65e2cd1a21d9" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 36551 } }
I20260925 22:36:22.763087 12645 raft_consensus.cc:374] T 00000000000000000000000000000000 P 76c34f8036954297be2d65e2cd1a21d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:36:22.763180 12645 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76c34f8036954297be2d65e2cd1a21d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c34f8036954297be2d65e2cd1a21d9, State: Initialized, Role: FOLLOWER
I20260925 22:36:22.763616 12645 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76c34f8036954297be2d65e2cd1a21d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76c34f8036954297be2d65e2cd1a21d9" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 36551 } }
I20260925 22:36:22.763767 12645 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76c34f8036954297be2d65e2cd1a21d9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:36:22.763855 12645 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76c34f8036954297be2d65e2cd1a21d9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:36:22.763942 12648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:22.755990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44064 (local address 127.6.246.254:36551)
0925 22:36:22.756642 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:22.756657 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:22.756690 (+ 33us) server_negotiation.cc:407] Connection header received
0925 22:36:22.757360 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:22.757378 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:22.757834 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:22.758166 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:22.758580 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:22.759568 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:22.760507 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:22.760985 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:22.761166 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:22.762019 (+ 853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:22.762025 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:22.762036 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:36:22.762408 (+ 372us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:22.762461 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:36:22.762480 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:22.762584 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:22.762748 (+ 164us) server_negotiation.cc:299] Negotiation successful
0925 22:36:22.763000 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":134,"threads_started":1}
I20260925 22:36:22.763952 12645 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76c34f8036954297be2d65e2cd1a21d9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:36:22.764987 12645 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76c34f8036954297be2d65e2cd1a21d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76c34f8036954297be2d65e2cd1a21d9" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 36551 } }
I20260925 22:36:22.765321 12645 leader_election.cc:302] T 00000000000000000000000000000000 P 76c34f8036954297be2d65e2cd1a21d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76c34f8036954297be2d65e2cd1a21d9; no voters:
I20260925 22:36:22.765646 12645 leader_election.cc:288] T 00000000000000000000000000000000 P 76c34f8036954297be2d65e2cd1a21d9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:36:22.765717 12650 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76c34f8036954297be2d65e2cd1a21d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:36:22.765967 12650 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76c34f8036954297be2d65e2cd1a21d9 [term 1 LEADER]: Becoming Leader. State: Replica: 76c34f8036954297be2d65e2cd1a21d9, State: Running, Role: LEADER
I20260925 22:36:22.766259 12650 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76c34f8036954297be2d65e2cd1a21d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76c34f8036954297be2d65e2cd1a21d9" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 36551 } }
I20260925 22:36:22.766742 7131 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:36:22.766942 12645 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76c34f8036954297be2d65e2cd1a21d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:36:22.767807 12652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76c34f8036954297be2d65e2cd1a21d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76c34f8036954297be2d65e2cd1a21d9. Latest consensus state: current_term: 1 leader_uuid: "76c34f8036954297be2d65e2cd1a21d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76c34f8036954297be2d65e2cd1a21d9" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 36551 } } }
I20260925 22:36:22.768080 12651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76c34f8036954297be2d65e2cd1a21d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "76c34f8036954297be2d65e2cd1a21d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76c34f8036954297be2d65e2cd1a21d9" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 36551 } } }
I20260925 22:36:22.768194 12652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76c34f8036954297be2d65e2cd1a21d9 [sys.catalog]: This master's current role is: LEADER
I20260925 22:36:22.768237 12651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76c34f8036954297be2d65e2cd1a21d9 [sys.catalog]: This master's current role is: LEADER
I20260925 22:36:22.768734 12659 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:36:22.771932 12659 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:36:22.777033 12659 catalog_manager.cc:1347] Generated new cluster ID: 05d94bdf97494a6daf27bcf2c26380b3
I20260925 22:36:22.777099 12659 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:36:22.797703 12659 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:36:22.798625 12659 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:36:22.805675 12659 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 76c34f8036954297be2d65e2cd1a21d9: Generated new TSK 0
I20260925 22:36:22.806337 12659 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:36:22.914604 12648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:22.814257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44080 (local address 127.6.246.254:36551)
0925 22:36:22.814382 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:22.814387 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:22.814567 (+ 180us) server_negotiation.cc:407] Connection header received
0925 22:36:22.814769 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:22.814771 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:22.814821 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:22.814964 (+ 143us) server_negotiation.cc:226] Negotiated authn=JWT
0925 22:36:22.815203 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:22.815921 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:22.816922 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:22.817157 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:22.817264 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:22.817568 (+ 304us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 22:36:22.914134 (+ 96566us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 22:36:22.914446 (+ 312us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44080: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":45}
W20260925 22:36:22.914737 12648 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44080: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260925 22:36:22.914772 12674 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:36:22.814168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:36551 (local address 127.0.0.1:44080)
0925 22:36:22.814478 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:22.814490 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:22.814617 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:22.814965 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:22.814968 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:22.814976 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0925 22:36:22.815091 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:22.815105 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:22.816037 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:22.816041 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:22.816799 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:22.816808 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:22.817272 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:22.817276 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:22.817377 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:22.817399 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 22:36:22.914352 (+ 96953us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 22:36:22.914541 (+ 189us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.246.254:36551: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":111,"threads_started":1}
I20260925 22:36:22.915458 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 12578
2026-09-25T22:36:22Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (374 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (120 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (119 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:36:23 dist-test-slave-wjdr krb5kdc[12686](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:36:23 dist-test-slave-wjdr krb5kdc[12686](info): set up 2 sockets
Sep 25 22:36:23 dist-test-slave-wjdr krb5kdc[12686](info): commencing operation
krb5kdc: starting...
W20260925 22:36:24.192960 7131 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Sep 25 22:36:24 dist-test-slave-wjdr krb5kdc[12686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375784, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:36:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:36:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.246.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab.
Entry for principal kudu/127.6.246.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab.
I20260925 22:36:24.312704 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:44345
--webserver_interface=127.6.246.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab
--principal=kudu/127.6.246.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:46265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.246.254:44345
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:36:24.421401 12699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:36:24.421640 12699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:36:24.421680 12699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:36:24.425592 12699 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:36:24.425648 12699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:36:24.425668 12699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:36:24.425684 12699 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:36:24.425699 12699 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:36:24.430874 12699 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:46265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.246.254:44345
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab
--principal=kudu/127.6.246.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:44345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.246.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12699
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:36:24.431787 12699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:36:24.432775 12699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 22:36:24.441143 12699 server_base.cc:1034] running on GCE node
W20260925 22:36:24.441071 12705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:24.441071 12707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:24.441089 12704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:24.441933 12699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:36:24.442963 12699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:36:24.444121 12699 hybrid_clock.cc:648] HybridClock initialized: now 1790375784444103 us; error 31 us; skew 500 ppm
Sep 25 22:36:24 dist-test-slave-wjdr krb5kdc[12686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375784, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:36:24.446924 12699 init.cc:377] Logged in from keytab as kudu/127.6.246.254@KRBTEST.COM (short username kudu)
I20260925 22:36:24.447779 12699 webserver.cc:460] Webserver started at http://127.6.246.254:45753/ using document root <none> and password file <none>
I20260925 22:36:24.448328 12699 fs_manager.cc:359] Metadata directory not provided
I20260925 22:36:24.448391 12699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:36:24.448649 12699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:36:24.450637 12699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/minicluster-data/master-0/data/instance:
uuid: "892287c6441e46998b2c718cae969392"
format_stamp: "Formatted at 2026-09-25 22:36:24 on dist-test-slave-wjdr"
server_key: "2c75dbe202c8b8146be613dd2ac79023"
server_key_iv: "896219db48a3812dd75029141b8ca11d"
server_key_version: "encryptionkey@0"
I20260925 22:36:24.451200 12699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/minicluster-data/master-0/wal/instance:
uuid: "892287c6441e46998b2c718cae969392"
format_stamp: "Formatted at 2026-09-25 22:36:24 on dist-test-slave-wjdr"
server_key: "2c75dbe202c8b8146be613dd2ac79023"
server_key_iv: "896219db48a3812dd75029141b8ca11d"
server_key_version: "encryptionkey@0"
I20260925 22:36:24.454795 12699 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 22:36:24.457068 12714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:24.458235 12699 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:36:24.458376 12699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/minicluster-data/master-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/minicluster-data/master-0/wal
uuid: "892287c6441e46998b2c718cae969392"
format_stamp: "Formatted at 2026-09-25 22:36:24 on dist-test-slave-wjdr"
server_key: "2c75dbe202c8b8146be613dd2ac79023"
server_key_iv: "896219db48a3812dd75029141b8ca11d"
server_key_version: "encryptionkey@0"
I20260925 22:36:24.458489 12699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:36:24.479418 12699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:36:24.483112 12699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:36:24.483315 12699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:36:24.490671 12699 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.254:44345
I20260925 22:36:24.490751 12766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.254:44345 every 8 connection(s)
I20260925 22:36:24.491591 12699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
I20260925 22:36:24.494359 12767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:36:24.498656 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 12699
I20260925 22:36:24.498791 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/minicluster-data/master-0/wal/instance
I20260925 22:36:24.499083 7131 external_mini_cluster.cc:1428] Setting key 065ff1c828e2923e41cc39f700edba09
I20260925 22:36:24.500869 12767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 892287c6441e46998b2c718cae969392: Bootstrap starting.
I20260925 22:36:24.503332 12767 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 892287c6441e46998b2c718cae969392: Neither blocks nor log segments found. Creating new log.
I20260925 22:36:24.504046 12767 log.cc:824] T 00000000000000000000000000000000 P 892287c6441e46998b2c718cae969392: Log is configured to *not* fsync() on all Append() calls
I20260925 22:36:24.506178 12767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 892287c6441e46998b2c718cae969392: No bootstrap required, opened a new log
Sep 25 22:36:24 dist-test-slave-wjdr krb5kdc[12686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375784, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
I20260925 22:36:24.510015 12767 raft_consensus.cc:348] T 00000000000000000000000000000000 P 892287c6441e46998b2c718cae969392 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "892287c6441e46998b2c718cae969392" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 44345 } }
I20260925 22:36:24.510214 12767 raft_consensus.cc:374] T 00000000000000000000000000000000 P 892287c6441e46998b2c718cae969392 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:36:24.510246 12767 raft_consensus.cc:729] T 00000000000000000000000000000000 P 892287c6441e46998b2c718cae969392 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 892287c6441e46998b2c718cae969392, State: Initialized, Role: FOLLOWER
I20260925 22:36:24.510677 12767 consensus_queue.cc:260] T 00000000000000000000000000000000 P 892287c6441e46998b2c718cae969392 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "892287c6441e46998b2c718cae969392" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 44345 } }
I20260925 22:36:24.510824 12767 raft_consensus.cc:388] T 00000000000000000000000000000000 P 892287c6441e46998b2c718cae969392 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:36:24.510903 12767 raft_consensus.cc:482] T 00000000000000000000000000000000 P 892287c6441e46998b2c718cae969392 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:36:24.511019 12767 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 892287c6441e46998b2c718cae969392 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:36:24.511837 12767 raft_consensus.cc:504] T 00000000000000000000000000000000 P 892287c6441e46998b2c718cae969392 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "892287c6441e46998b2c718cae969392" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 44345 } }
I20260925 22:36:24.512156 12767 leader_election.cc:302] T 00000000000000000000000000000000 P 892287c6441e46998b2c718cae969392 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 892287c6441e46998b2c718cae969392; no voters:
I20260925 22:36:24.512524 12767 leader_election.cc:288] T 00000000000000000000000000000000 P 892287c6441e46998b2c718cae969392 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:36:24.512599 12772 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 892287c6441e46998b2c718cae969392 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:36:24.512840 12772 raft_consensus.cc:686] T 00000000000000000000000000000000 P 892287c6441e46998b2c718cae969392 [term 1 LEADER]: Becoming Leader. State: Replica: 892287c6441e46998b2c718cae969392, State: Running, Role: LEADER
I20260925 22:36:24.513130 12772 consensus_queue.cc:237] T 00000000000000000000000000000000 P 892287c6441e46998b2c718cae969392 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "892287c6441e46998b2c718cae969392" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 44345 } }
I20260925 22:36:24.513513 12770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:24.500434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58526 (local address 127.6.246.254:44345)
0925 22:36:24.500917 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:24.500941 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:24.500985 (+ 44us) server_negotiation.cc:407] Connection header received
0925 22:36:24.501623 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:24.501639 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:24.501950 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:24.502221 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:24.503130 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:24.504221 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:24.505355 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:24.505804 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:24.505994 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:24.508709 (+ 2715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:24.508715 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:24.508726 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:24.508751 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:24.510531 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:24.510968 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:24.510974 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:24.510981 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:24.511139 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:24.511421 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:24.511424 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:24.511426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:24.511788 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:24.512023 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:24.512254 (+ 231us) server_negotiation.cc:299] Negotiation successful
0925 22:36:24.512574 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":129,"threads_started":1}
I20260925 22:36:24.513765 12767 sys_catalog.cc:564] T 00000000000000000000000000000000 P 892287c6441e46998b2c718cae969392 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:36:24.514288 12772 sys_catalog.cc:455] T 00000000000000000000000000000000 P 892287c6441e46998b2c718cae969392 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 892287c6441e46998b2c718cae969392. Latest consensus state: current_term: 1 leader_uuid: "892287c6441e46998b2c718cae969392" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "892287c6441e46998b2c718cae969392" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 44345 } } }
I20260925 22:36:24.514407 12772 sys_catalog.cc:458] T 00000000000000000000000000000000 P 892287c6441e46998b2c718cae969392 [sys.catalog]: This master's current role is: LEADER
I20260925 22:36:24.514698 12773 sys_catalog.cc:455] T 00000000000000000000000000000000 P 892287c6441e46998b2c718cae969392 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "892287c6441e46998b2c718cae969392" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "892287c6441e46998b2c718cae969392" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 44345 } } }
I20260925 22:36:24.514834 12773 sys_catalog.cc:458] T 00000000000000000000000000000000 P 892287c6441e46998b2c718cae969392 [sys.catalog]: This master's current role is: LEADER
W20260925 22:36:24.518260 12787 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 892287c6441e46998b2c718cae969392: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:36:24.518323 12787 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:36:24.518409 12776 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:36:24.519205 12776 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:36:24.523974 12776 catalog_manager.cc:1347] Generated new cluster ID: aa23e76d48b846f1b426390585a099d8
I20260925 22:36:24.524032 12776 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:36:24.535061 12776 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:36:24.535977 12776 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:36:24.545773 12776 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 892287c6441e46998b2c718cae969392: Generated new TSK 0
I20260925 22:36:24.546473 12776 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.246.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.193.keytab.
Entry for principal kudu/127.6.246.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.193.keytab.
I20260925 22:36:24.579977 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.193:0
--local_ip_for_outbound_sockets=127.6.246.193
--webserver_interface=127.6.246.193
--webserver_port=0
--tserver_master_addrs=127.6.246.254:44345
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.193.keytab
--principal=kudu/127.6.246.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:46265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:36:24.688876 12793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:36:24.689168 12793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:36:24.689210 12793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:36:24.693320 12793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:36:24.693370 12793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:36:24.693431 12793 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.193
I20260925 22:36:24.698572 12793 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:46265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.193.keytab
--principal=kudu/127.6.246.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.246.193
--webserver_port=0
--tserver_master_addrs=127.6.246.254:44345
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12793
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.193
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:36:24.699455 12793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:36:24.700408 12793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:36:24.709754 12798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:24.709739 12793 server_base.cc:1034] running on GCE node
W20260925 22:36:24.709671 12799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:24.709672 12801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:24.710366 12793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:36:24.710968 12793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:36:24.712155 12793 hybrid_clock.cc:648] HybridClock initialized: now 1790375784712125 us; error 42 us; skew 500 ppm
Sep 25 22:36:24 dist-test-slave-wjdr krb5kdc[12686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375784, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:36:24.715098 12793 init.cc:377] Logged in from keytab as kudu/127.6.246.193@KRBTEST.COM (short username kudu)
I20260925 22:36:24.715909 12793 webserver.cc:460] Webserver started at http://127.6.246.193:33339/ using document root <none> and password file <none>
I20260925 22:36:24.716495 12793 fs_manager.cc:359] Metadata directory not provided
I20260925 22:36:24.716594 12793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:36:24.716799 12793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:36:24.718737 12793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/minicluster-data/ts-0/data/instance:
uuid: "d908ba8c21db414db33122b8dd2aa818"
format_stamp: "Formatted at 2026-09-25 22:36:24 on dist-test-slave-wjdr"
server_key: "9eda70a6d1f2e205f5a4852c11363e22"
server_key_iv: "5c32ae3a7b3f4f070713c7b6d21d74f1"
server_key_version: "encryptionkey@0"
I20260925 22:36:24.719270 12793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/minicluster-data/ts-0/wal/instance:
uuid: "d908ba8c21db414db33122b8dd2aa818"
format_stamp: "Formatted at 2026-09-25 22:36:24 on dist-test-slave-wjdr"
server_key: "9eda70a6d1f2e205f5a4852c11363e22"
server_key_iv: "5c32ae3a7b3f4f070713c7b6d21d74f1"
server_key_version: "encryptionkey@0"
I20260925 22:36:24.722708 12793 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 22:36:24.725090 12808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:24.726191 12793 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:36:24.726331 12793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/minicluster-data/ts-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/minicluster-data/ts-0/wal
uuid: "d908ba8c21db414db33122b8dd2aa818"
format_stamp: "Formatted at 2026-09-25 22:36:24 on dist-test-slave-wjdr"
server_key: "9eda70a6d1f2e205f5a4852c11363e22"
server_key_iv: "5c32ae3a7b3f4f070713c7b6d21d74f1"
server_key_version: "encryptionkey@0"
I20260925 22:36:24.726432 12793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:36:24.742465 12793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:36:24.745766 12793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:36:24.745994 12793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:36:24.746797 12793 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:36:24.747716 12793 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:36:24.747783 12793 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:24.747856 12793 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:36:24.747886 12793 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:24.756645 12793 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.193:43081
I20260925 22:36:24.756709 12921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.193:43081 every 8 connection(s)
I20260925 22:36:24.757543 12793 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
I20260925 22:36:24.765933 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 12793
I20260925 22:36:24.766070 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/minicluster-data/ts-0/wal/instance
I20260925 22:36:24.766357 7131 external_mini_cluster.cc:1428] Setting key b4f05a8cfbd8c82fdf8eaf063b1c1408
Sep 25 22:36:24 dist-test-slave-wjdr krb5kdc[12686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375784, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.193@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
I20260925 22:36:24.771976 12770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:24.759258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.193:58743 (local address 127.6.246.254:44345)
0925 22:36:24.759402 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:24.759407 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:24.760426 (+ 1019us) server_negotiation.cc:407] Connection header received
0925 22:36:24.761967 (+ 1541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:24.761970 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:24.762016 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:24.762095 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:24.763277 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:24.763810 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:24.765015 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:24.765224 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:24.765300 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:24.768020 (+ 2720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:24.768024 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:24.768026 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:24.768049 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:24.769607 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:24.770260 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:24.770262 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:24.770264 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:24.770347 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:24.770745 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:24.770748 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:24.770749 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:24.770979 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:24.771061 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:24.771584 (+ 523us) server_negotiation.cc:299] Negotiation successful
0925 22:36:24.771731 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 22:36:24.772682 12924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:24.759705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:44345 (local address 127.6.246.193:58743)
0925 22:36:24.760235 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:24.760289 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:24.761753 (+ 1464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:24.762248 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:24.762255 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:24.762697 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:24.763151 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:24.763164 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:24.763936 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:24.763940 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:24.764917 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:24.764926 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:24.765352 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:24.765355 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:24.765486 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:24.766161 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:24.766184 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:24.767844 (+ 1660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:36:24.769725 (+ 1881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:24.769733 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:24.769750 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:24.770141 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:24.770437 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:24.770443 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:24.770446 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:24.770645 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:24.771072 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:36:24.771077 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:36:24.771334 (+ 257us) client_negotiation.cc:769] Sending connection context
0925 22:36:24.771497 (+ 163us) client_negotiation.cc:240] Negotiation successful
0925 22:36:24.771753 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":114,"threads_started":1}
I20260925 22:36:24.773936 12922 heartbeater.cc:347] Connected to a master server at 127.6.246.254:44345
I20260925 22:36:24.774163 12922 heartbeater.cc:464] Registering TS with master...
I20260925 22:36:24.774636 12922 heartbeater.cc:511] Master 127.6.246.254:44345 requested a full tablet report, sending...
I20260925 22:36:24.776084 12731 ts_manager.cc:194] Registered new tserver with Master: d908ba8c21db414db33122b8dd2aa818 (127.6.246.193:43081)
WARNING: no policy specified for kudu/127.6.246.194@KRBTEST.COM; defaulting to no policy
I20260925 22:36:24.777268 12731 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.246.193@KRBTEST.COM'} at 127.6.246.193:58743
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.194.keytab.
Entry for principal kudu/127.6.246.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.194.keytab.
I20260925 22:36:24.793682 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.194:0
--local_ip_for_outbound_sockets=127.6.246.194
--webserver_interface=127.6.246.194
--webserver_port=0
--tserver_master_addrs=127.6.246.254:44345
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.194.keytab
--principal=kudu/127.6.246.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:46265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:36:24.903007 12927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:36:24.903228 12927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:36:24.903296 12927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:36:24.907301 12927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:36:24.907348 12927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:36:24.907430 12927 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.194
I20260925 22:36:24.912392 12927 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:46265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.194.keytab
--principal=kudu/127.6.246.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.246.194
--webserver_port=0
--tserver_master_addrs=127.6.246.254:44345
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12927
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.194
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:36:24.913381 12927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:36:24.914351 12927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:36:24.923592 12933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:24.923602 12932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:24.923610 12935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:24.923741 12927 server_base.cc:1034] running on GCE node
I20260925 22:36:24.924266 12927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:36:24.924906 12927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:36:24.926054 12927 hybrid_clock.cc:648] HybridClock initialized: now 1790375784926026 us; error 42 us; skew 500 ppm
Sep 25 22:36:24 dist-test-slave-wjdr krb5kdc[12686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375784, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:36:24.928635 12927 init.cc:377] Logged in from keytab as kudu/127.6.246.194@KRBTEST.COM (short username kudu)
I20260925 22:36:24.929425 12927 webserver.cc:460] Webserver started at http://127.6.246.194:35055/ using document root <none> and password file <none>
I20260925 22:36:24.929944 12927 fs_manager.cc:359] Metadata directory not provided
I20260925 22:36:24.930001 12927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:36:24.930204 12927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:36:24.932126 12927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/minicluster-data/ts-1/data/instance:
uuid: "cdcdb98addbb46cd8501e5f75d86efca"
format_stamp: "Formatted at 2026-09-25 22:36:24 on dist-test-slave-wjdr"
server_key: "898db2df1bcf0eadd9d42a824e997d39"
server_key_iv: "7c0776c9cb63492f2bffd53dd95ded93"
server_key_version: "encryptionkey@0"
I20260925 22:36:24.932672 12927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/minicluster-data/ts-1/wal/instance:
uuid: "cdcdb98addbb46cd8501e5f75d86efca"
format_stamp: "Formatted at 2026-09-25 22:36:24 on dist-test-slave-wjdr"
server_key: "898db2df1bcf0eadd9d42a824e997d39"
server_key_iv: "7c0776c9cb63492f2bffd53dd95ded93"
server_key_version: "encryptionkey@0"
I20260925 22:36:24.936012 12927 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260925 22:36:24.938205 12942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:24.939289 12927 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:36:24.939384 12927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/minicluster-data/ts-1/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/minicluster-data/ts-1/wal
uuid: "cdcdb98addbb46cd8501e5f75d86efca"
format_stamp: "Formatted at 2026-09-25 22:36:24 on dist-test-slave-wjdr"
server_key: "898db2df1bcf0eadd9d42a824e997d39"
server_key_iv: "7c0776c9cb63492f2bffd53dd95ded93"
server_key_version: "encryptionkey@0"
I20260925 22:36:24.939458 12927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:36:24.953634 12927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:36:24.956261 12927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:36:24.956425 12927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:36:24.956959 12927 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:36:24.957679 12927 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:36:24.957724 12927 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:24.957760 12927 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:36:24.957780 12927 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:24.966485 12927 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.194:46491
I20260925 22:36:24.966557 13055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.194:46491 every 8 connection(s)
I20260925 22:36:24.967336 12927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
I20260925 22:36:24.969447 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 12927
I20260925 22:36:24.969580 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/minicluster-data/ts-1/wal/instance
I20260925 22:36:24.969935 7131 external_mini_cluster.cc:1428] Setting key a3a798f531e52487f3fe00a864b35713
Sep 25 22:36:24 dist-test-slave-wjdr krb5kdc[12686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375784, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.194@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.246.195@KRBTEST.COM; defaulting to no policy
I20260925 22:36:24.981855 12770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:24.969050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.194:56533 (local address 127.6.246.254:44345)
0925 22:36:24.969173 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:24.969177 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:24.970046 (+ 869us) server_negotiation.cc:407] Connection header received
0925 22:36:24.971496 (+ 1450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:24.971499 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:24.971542 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:24.971607 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:24.973218 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:24.974109 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:24.975263 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:24.975722 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:24.975843 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:24.978227 (+ 2384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:24.978231 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:24.978234 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:24.978264 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:24.979896 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:24.980350 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:24.980353 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:24.980354 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:24.980441 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:24.980836 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:24.980840 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:24.980842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:24.981039 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:24.981122 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:24.981635 (+ 513us) server_negotiation.cc:299] Negotiation successful
0925 22:36:24.981734 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.195@KRBTEST.COM" created.
I20260925 22:36:24.982906 13058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:24.969357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:44345 (local address 127.6.246.194:56533)
0925 22:36:24.969887 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:24.969941 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:24.971273 (+ 1332us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:24.971793 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:24.971804 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:24.972340 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:24.973035 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:24.973057 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:24.974244 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:24.974246 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:24.975139 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:24.975148 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:24.975867 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:24.975871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:24.976010 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:24.976681 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:24.976696 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:24.978064 (+ 1368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:36:24.980014 (+ 1950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:24.980018 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:24.980029 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:24.980262 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:24.980565 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:24.980568 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:24.980569 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:24.980741 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:24.981151 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:36:24.981156 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:36:24.981388 (+ 232us) client_negotiation.cc:769] Sending connection context
0925 22:36:24.981568 (+ 180us) client_negotiation.cc:240] Negotiation successful
0925 22:36:24.981837 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":148,"threads_started":1}
I20260925 22:36:24.984074 13056 heartbeater.cc:347] Connected to a master server at 127.6.246.254:44345
I20260925 22:36:24.984308 13056 heartbeater.cc:464] Registering TS with master...
I20260925 22:36:24.984800 13056 heartbeater.cc:511] Master 127.6.246.254:44345 requested a full tablet report, sending...
I20260925 22:36:24.985831 12731 ts_manager.cc:194] Registered new tserver with Master: cdcdb98addbb46cd8501e5f75d86efca (127.6.246.194:46491)
I20260925 22:36:24.986676 12731 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.246.194@KRBTEST.COM'} at 127.6.246.194:56533
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.195.keytab.
Entry for principal kudu/127.6.246.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.195.keytab.
I20260925 22:36:24.996924 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.195:0
--local_ip_for_outbound_sockets=127.6.246.195
--webserver_interface=127.6.246.195
--webserver_port=0
--tserver_master_addrs=127.6.246.254:44345
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.195.keytab
--principal=kudu/127.6.246.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:46265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:36:25.104162 13061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:36:25.104410 13061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:36:25.104497 13061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:36:25.108589 13061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:36:25.108644 13061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:36:25.108700 13061 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.195
I20260925 22:36:25.113812 13061 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:46265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.195.keytab
--principal=kudu/127.6.246.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.246.195
--webserver_port=0
--tserver_master_addrs=127.6.246.254:44345
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13061
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.195
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:36:25.114822 13061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:36:25.115850 13061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:36:25.125420 13066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:25.125439 13069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:25.125416 13067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:25.125639 13061 server_base.cc:1034] running on GCE node
I20260925 22:36:25.126132 13061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:36:25.126741 13061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:36:25.127923 13061 hybrid_clock.cc:648] HybridClock initialized: now 1790375785127914 us; error 32 us; skew 500 ppm
Sep 25 22:36:25 dist-test-slave-wjdr krb5kdc[12686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375785, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:36:25.130693 13061 init.cc:377] Logged in from keytab as kudu/127.6.246.195@KRBTEST.COM (short username kudu)
I20260925 22:36:25.131513 13061 webserver.cc:460] Webserver started at http://127.6.246.195:44213/ using document root <none> and password file <none>
I20260925 22:36:25.132047 13061 fs_manager.cc:359] Metadata directory not provided
I20260925 22:36:25.132129 13061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:36:25.132340 13061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:36:25.134341 13061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/minicluster-data/ts-2/data/instance:
uuid: "f5822dd76fee422c8a66da7f6e6ed88d"
format_stamp: "Formatted at 2026-09-25 22:36:25 on dist-test-slave-wjdr"
server_key: "01f935ffc0b008ce991d0cabbd418a4c"
server_key_iv: "ec7ef2a0a15981cbe55c2bfbbca40559"
server_key_version: "encryptionkey@0"
I20260925 22:36:25.134883 13061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/minicluster-data/ts-2/wal/instance:
uuid: "f5822dd76fee422c8a66da7f6e6ed88d"
format_stamp: "Formatted at 2026-09-25 22:36:25 on dist-test-slave-wjdr"
server_key: "01f935ffc0b008ce991d0cabbd418a4c"
server_key_iv: "ec7ef2a0a15981cbe55c2bfbbca40559"
server_key_version: "encryptionkey@0"
I20260925 22:36:25.138299 13061 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 22:36:25.140511 13076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:25.141607 13061 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:36:25.141731 13061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/minicluster-data/ts-2/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/minicluster-data/ts-2/wal
uuid: "f5822dd76fee422c8a66da7f6e6ed88d"
format_stamp: "Formatted at 2026-09-25 22:36:25 on dist-test-slave-wjdr"
server_key: "01f935ffc0b008ce991d0cabbd418a4c"
server_key_iv: "ec7ef2a0a15981cbe55c2bfbbca40559"
server_key_version: "encryptionkey@0"
I20260925 22:36:25.141826 13061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:36:25.156942 13061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:36:25.159701 13061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:36:25.159904 13061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:36:25.160485 13061 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:36:25.161283 13061 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:36:25.161348 13061 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:25.161407 13061 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:36:25.161458 13061 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:25.170150 13061 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.195:40671
I20260925 22:36:25.170172 13189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.195:40671 every 8 connection(s)
I20260925 22:36:25.170995 13061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
I20260925 22:36:25.172502 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 13061
I20260925 22:36:25.172621 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/minicluster-data/ts-2/wal/instance
I20260925 22:36:25.172924 7131 external_mini_cluster.cc:1428] Setting key 2bd31fd5ea9a22e4b3372681976ba066
Sep 25 22:36:25 dist-test-slave-wjdr krb5kdc[12686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375785, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.195@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
I20260925 22:36:25.183961 12770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:25.172747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.195:56785 (local address 127.6.246.254:44345)
0925 22:36:25.172903 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:25.172907 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:25.173872 (+ 965us) server_negotiation.cc:407] Connection header received
0925 22:36:25.175257 (+ 1385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:25.175260 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:25.175304 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:25.175371 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:25.176433 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:25.177004 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:25.178101 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:25.178307 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:25.178396 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:25.180812 (+ 2416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:25.180815 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:25.180817 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:25.180842 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:25.182100 (+ 1258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:25.182549 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:25.182552 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:25.182553 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:25.182623 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:25.182907 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:25.182909 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:25.182911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:25.183076 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:25.183153 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:25.183720 (+ 567us) server_negotiation.cc:299] Negotiation successful
0925 22:36:25.183830 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 22:36:25.184851 13192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:25.173232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:44345 (local address 127.6.246.195:56785)
0925 22:36:25.173701 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:25.173782 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:25.175080 (+ 1298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:25.175518 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:25.175524 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:25.175854 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:25.176294 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:25.176306 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:25.177118 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:25.177120 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:25.178001 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:25.178010 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:25.178425 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:25.178428 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:25.178570 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:25.179194 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:25.179210 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:25.180657 (+ 1447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:36:25.182200 (+ 1543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:25.182204 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:25.182216 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:25.182465 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:25.182696 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:25.182699 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:25.182700 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:25.182827 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:25.183197 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:36:25.183202 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:36:25.183455 (+ 253us) client_negotiation.cc:769] Sending connection context
0925 22:36:25.183634 (+ 179us) client_negotiation.cc:240] Negotiation successful
0925 22:36:25.183895 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":131,"threads_started":1}
I20260925 22:36:25.185992 13190 heartbeater.cc:347] Connected to a master server at 127.6.246.254:44345
I20260925 22:36:25.186239 13190 heartbeater.cc:464] Registering TS with master...
I20260925 22:36:25.186764 13190 heartbeater.cc:511] Master 127.6.246.254:44345 requested a full tablet report, sending...
I20260925 22:36:25.187718 12730 ts_manager.cc:194] Registered new tserver with Master: f5822dd76fee422c8a66da7f6e6ed88d (127.6.246.195:40671)
I20260925 22:36:25.188423 12730 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.246.195@KRBTEST.COM'} at 127.6.246.195:56785
I20260925 22:36:25.198117 7131 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/krb5kdc/test-admin.keytab.
Sep 25 22:36:25 dist-test-slave-wjdr krb5kdc[12686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:36:25.212275 7131 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260925 22:36:25.215021 13197 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:36:25.213533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44142 (local address 127.6.246.193:43081)
0925 22:36:25.213889 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:25.213899 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:25.213921 (+ 22us) server_negotiation.cc:407] Connection header received
0925 22:36:25.213967 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:25.213972 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:25.214141 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:25.214318 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:25.214836 (+ 518us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44142: BlockingRecv error: recv got EOF from 127.0.0.1:44142 (error 108)
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":121,"threads_started":1}
W20260925 22:36:25.215034 7131 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/krb5kdc/krb5cc' of type FILE
Sep 25 22:36:25 dist-test-slave-wjdr krb5kdc[12686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:36:25.216030 7131 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 22:36:25 dist-test-slave-wjdr krb5kdc[12686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:36:25.216977 7131 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 22:36:25 dist-test-slave-wjdr krb5kdc[12686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.246.193@KRBTEST.COM
I20260925 22:36:25.225955 13197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:25.217748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44150 (local address 127.6.246.193:43081)
0925 22:36:25.217867 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:25.217871 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:25.217949 (+ 78us) server_negotiation.cc:407] Connection header received
0925 22:36:25.218082 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:25.218089 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:25.218123 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:25.218203 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:25.218718 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:25.219464 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:25.220300 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:25.220716 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:25.220828 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:25.223033 (+ 2205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:25.223049 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:25.223060 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:25.223081 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:25.224519 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:25.224899 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:25.224904 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:25.224905 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:25.224978 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:25.225189 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:25.225191 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:25.225192 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:25.225414 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:25.225575 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:25.225716 (+ 141us) server_negotiation.cc:299] Negotiation successful
0925 22:36:25.225829 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260925 22:36:25.229444 13197 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:36:25.228510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44154 (local address 127.6.246.193:43081)
0925 22:36:25.228605 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:25.228608 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:25.228692 (+ 84us) server_negotiation.cc:407] Connection header received
0925 22:36:25.228816 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:25.228819 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:25.228861 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:25.228953 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:25.229345 (+ 392us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44154: BlockingRecv error: recv got EOF from 127.0.0.1:44154 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260925 22:36:25.229573 7131 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/krb5kdc/krb5cc' of type FILE
Sep 25 22:36:25 dist-test-slave-wjdr krb5kdc[12686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:36:25.230535 7131 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 22:36:25 dist-test-slave-wjdr krb5kdc[12686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:36:25.231408 7131 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 22:36:25 dist-test-slave-wjdr krb5kdc[12686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.246.193@KRBTEST.COM
I20260925 22:36:25.239460 13197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:25.232089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44166 (local address 127.6.246.193:43081)
0925 22:36:25.232197 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:25.232203 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:25.232329 (+ 126us) server_negotiation.cc:407] Connection header received
0925 22:36:25.232372 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:25.232374 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:25.232407 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:25.232581 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:25.233049 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:25.233586 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:25.234327 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:25.234539 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:25.234615 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:25.236645 (+ 2030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:25.236649 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:25.236651 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:25.236670 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:25.237939 (+ 1269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:25.238339 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:25.238343 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:25.238344 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:25.238445 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:25.238703 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:25.238706 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:25.238707 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:25.238874 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:25.238979 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:25.239162 (+ 183us) server_negotiation.cc:299] Negotiation successful
0925 22:36:25.239287 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260925 22:36:25.241988 13197 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:36:25.241191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44174 (local address 127.6.246.193:43081)
0925 22:36:25.241314 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:25.241317 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:25.241338 (+ 21us) server_negotiation.cc:407] Connection header received
0925 22:36:25.241437 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:25.241439 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:25.241471 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:25.241532 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:25.241886 (+ 354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44174: BlockingRecv error: recv got EOF from 127.0.0.1:44174 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260925 22:36:25.242051 7131 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790375752797504-7131-0/krb5kdc/krb5cc' of type FILE
Sep 25 22:36:25 dist-test-slave-wjdr krb5kdc[12686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:36:25.243021 7131 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 22:36:25 dist-test-slave-wjdr krb5kdc[12686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:36:25.243867 7131 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 22:36:25 dist-test-slave-wjdr krb5kdc[12686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.246.193@KRBTEST.COM
I20260925 22:36:25.251984 13197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:25.244611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44176 (local address 127.6.246.193:43081)
0925 22:36:25.244733 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:25.244737 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:25.244851 (+ 114us) server_negotiation.cc:407] Connection header received
0925 22:36:25.245001 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:25.245004 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:25.245051 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:25.245116 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:25.245590 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:25.246070 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:25.246872 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:25.247099 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:25.247202 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:25.249311 (+ 2109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:25.249315 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:25.249317 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:25.249340 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:25.250500 (+ 1160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:25.250888 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:25.250894 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:25.250898 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:25.250976 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:25.251231 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:25.251235 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:25.251237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:25.251395 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:25.251523 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:25.251738 (+ 215us) server_negotiation.cc:299] Negotiation successful
0925 22:36:25.251845 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 22:36:25.252912 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 12793
I20260925 22:36:25.258589 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 12927
I20260925 22:36:25.263921 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 13061
I20260925 22:36:25.269512 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 12699
2026-09-25T22:36:25Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2115 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:36:25 dist-test-slave-wjdr krb5kdc[13210](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:36:25 dist-test-slave-wjdr krb5kdc[13210](info): set up 2 sockets
Sep 25 22:36:25 dist-test-slave-wjdr krb5kdc[13210](info): commencing operation
krb5kdc: starting...
W20260925 22:36:26.304942 7131 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Sep 25 22:36:26 dist-test-slave-wjdr krb5kdc[13210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375786, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:36:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:36:26Z Disabled control of system clock
WARNING: no policy specified for oryx/127.6.246.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.246.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.246.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/krb5kdc/oryx_127.6.246.254.keytab.
Entry for principal oryx/127.6.246.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/krb5kdc/oryx_127.6.246.254.keytab.
I20260925 22:36:26.427471 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:44671
--webserver_interface=127.6.246.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/krb5kdc/oryx_127.6.246.254.keytab
--principal=oryx/127.6.246.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:39015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.246.254:44671
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:36:26.537889 13223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:36:26.538143 13223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:36:26.538183 13223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:36:26.542205 13223 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:36:26.542265 13223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:36:26.542286 13223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:36:26.542302 13223 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:36:26.542317 13223 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:36:26.551110 13223 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:39015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.246.254:44671
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/krb5kdc/oryx_127.6.246.254.keytab
--principal=oryx/127.6.246.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:44671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.246.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13223
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:36:26.552140 13223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:36:26.553196 13223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 22:36:26.561545 13223 server_base.cc:1034] running on GCE node
W20260925 22:36:26.561514 13229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:26.561514 13228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:26.561707 13231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:26.562282 13223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:36:26.563401 13223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:36:26.564569 13223 hybrid_clock.cc:648] HybridClock initialized: now 1790375786564556 us; error 26 us; skew 500 ppm
Sep 25 22:36:26 dist-test-slave-wjdr krb5kdc[13210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375786, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.246.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:36:26.571365 13223 init.cc:377] Logged in from keytab as oryx/127.6.246.254@KRBTEST.COM (short username oryx)
I20260925 22:36:26.572268 13223 webserver.cc:460] Webserver started at http://127.6.246.254:41529/ using document root <none> and password file <none>
I20260925 22:36:26.572889 13223 fs_manager.cc:359] Metadata directory not provided
I20260925 22:36:26.572978 13223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:36:26.573197 13223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:36:26.575255 13223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/master-0/data/instance:
uuid: "f241ea7630914735a806c4b2935b467d"
format_stamp: "Formatted at 2026-09-25 22:36:26 on dist-test-slave-wjdr"
server_key: "3d7298dd0942972dfc60ed90bc80f2ea"
server_key_iv: "b1a975b07a8913afca562b450b77fdba"
server_key_version: "encryptionkey@0"
I20260925 22:36:26.575807 13223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/master-0/wal/instance:
uuid: "f241ea7630914735a806c4b2935b467d"
format_stamp: "Formatted at 2026-09-25 22:36:26 on dist-test-slave-wjdr"
server_key: "3d7298dd0942972dfc60ed90bc80f2ea"
server_key_iv: "b1a975b07a8913afca562b450b77fdba"
server_key_version: "encryptionkey@0"
I20260925 22:36:26.579439 13223 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 22:36:26.581954 13238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:26.583323 13223 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 22:36:26.583454 13223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/master-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/master-0/wal
uuid: "f241ea7630914735a806c4b2935b467d"
format_stamp: "Formatted at 2026-09-25 22:36:26 on dist-test-slave-wjdr"
server_key: "3d7298dd0942972dfc60ed90bc80f2ea"
server_key_iv: "b1a975b07a8913afca562b450b77fdba"
server_key_version: "encryptionkey@0"
I20260925 22:36:26.583564 13223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:36:26.627827 13223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:36:26.630652 13223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:36:26.630846 13223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:36:26.637487 13223 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.254:44671
I20260925 22:36:26.637512 13290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.254:44671 every 8 connection(s)
I20260925 22:36:26.638366 13223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
I20260925 22:36:26.640950 13291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:36:26.644112 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 13223
I20260925 22:36:26.644248 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/master-0/wal/instance
I20260925 22:36:26.644549 7131 external_mini_cluster.cc:1428] Setting key 1758b2f72368bd07d64ac7ba96aad8c0
I20260925 22:36:26.646638 13291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f241ea7630914735a806c4b2935b467d: Bootstrap starting.
I20260925 22:36:26.648697 13291 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f241ea7630914735a806c4b2935b467d: Neither blocks nor log segments found. Creating new log.
I20260925 22:36:26.649386 13291 log.cc:824] T 00000000000000000000000000000000 P f241ea7630914735a806c4b2935b467d: Log is configured to *not* fsync() on all Append() calls
I20260925 22:36:26.651079 13291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f241ea7630914735a806c4b2935b467d: No bootstrap required, opened a new log
Sep 25 22:36:26 dist-test-slave-wjdr krb5kdc[13210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375786, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM
I20260925 22:36:26.654994 13291 raft_consensus.cc:348] T 00000000000000000000000000000000 P f241ea7630914735a806c4b2935b467d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f241ea7630914735a806c4b2935b467d" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 44671 } }
I20260925 22:36:26.655201 13291 raft_consensus.cc:374] T 00000000000000000000000000000000 P f241ea7630914735a806c4b2935b467d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:36:26.655234 13291 raft_consensus.cc:729] T 00000000000000000000000000000000 P f241ea7630914735a806c4b2935b467d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f241ea7630914735a806c4b2935b467d, State: Initialized, Role: FOLLOWER
I20260925 22:36:26.655683 13291 consensus_queue.cc:260] T 00000000000000000000000000000000 P f241ea7630914735a806c4b2935b467d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f241ea7630914735a806c4b2935b467d" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 44671 } }
I20260925 22:36:26.655802 13291 raft_consensus.cc:388] T 00000000000000000000000000000000 P f241ea7630914735a806c4b2935b467d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:36:26.655880 13291 raft_consensus.cc:482] T 00000000000000000000000000000000 P f241ea7630914735a806c4b2935b467d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:36:26.655977 13291 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f241ea7630914735a806c4b2935b467d [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:36:26.657065 13291 raft_consensus.cc:504] T 00000000000000000000000000000000 P f241ea7630914735a806c4b2935b467d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f241ea7630914735a806c4b2935b467d" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 44671 } }
I20260925 22:36:26.657445 13291 leader_election.cc:302] T 00000000000000000000000000000000 P f241ea7630914735a806c4b2935b467d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f241ea7630914735a806c4b2935b467d; no voters:
I20260925 22:36:26.657770 13291 leader_election.cc:288] T 00000000000000000000000000000000 P f241ea7630914735a806c4b2935b467d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:36:26.657887 13296 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f241ea7630914735a806c4b2935b467d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:36:26.658113 13296 raft_consensus.cc:686] T 00000000000000000000000000000000 P f241ea7630914735a806c4b2935b467d [term 1 LEADER]: Becoming Leader. State: Replica: f241ea7630914735a806c4b2935b467d, State: Running, Role: LEADER
I20260925 22:36:26.658404 13296 consensus_queue.cc:237] T 00000000000000000000000000000000 P f241ea7630914735a806c4b2935b467d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f241ea7630914735a806c4b2935b467d" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 44671 } }
I20260925 22:36:26.658788 13294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:26.645949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50314 (local address 127.6.246.254:44671)
0925 22:36:26.646357 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:26.646371 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:26.646404 (+ 33us) server_negotiation.cc:407] Connection header received
0925 22:36:26.647078 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:26.647096 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:26.647393 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:26.647701 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:26.648510 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:26.649567 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:26.650440 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:26.650908 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:26.651068 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:26.653715 (+ 2647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:26.653720 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:26.653732 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:26.653777 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:26.655568 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:26.656137 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:26.656143 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:26.656148 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:26.656288 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:26.656601 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:26.656604 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:26.656606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:26.657126 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:26.657326 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:26.657597 (+ 271us) server_negotiation.cc:299] Negotiation successful
0925 22:36:26.657904 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":114,"threads_started":1}
I20260925 22:36:26.658969 13291 sys_catalog.cc:564] T 00000000000000000000000000000000 P f241ea7630914735a806c4b2935b467d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:36:26.660018 13298 sys_catalog.cc:455] T 00000000000000000000000000000000 P f241ea7630914735a806c4b2935b467d [sys.catalog]: SysCatalogTable state changed. Reason: New leader f241ea7630914735a806c4b2935b467d. Latest consensus state: current_term: 1 leader_uuid: "f241ea7630914735a806c4b2935b467d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f241ea7630914735a806c4b2935b467d" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 44671 } } }
I20260925 22:36:26.660151 13298 sys_catalog.cc:458] T 00000000000000000000000000000000 P f241ea7630914735a806c4b2935b467d [sys.catalog]: This master's current role is: LEADER
I20260925 22:36:26.660076 13297 sys_catalog.cc:455] T 00000000000000000000000000000000 P f241ea7630914735a806c4b2935b467d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f241ea7630914735a806c4b2935b467d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f241ea7630914735a806c4b2935b467d" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 44671 } } }
I20260925 22:36:26.660498 13297 sys_catalog.cc:458] T 00000000000000000000000000000000 P f241ea7630914735a806c4b2935b467d [sys.catalog]: This master's current role is: LEADER
I20260925 22:36:26.661208 13305 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:36:26.664197 13305 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:36:26.669790 13305 catalog_manager.cc:1347] Generated new cluster ID: 2f47f719071c41b3ad8263c885d14031
I20260925 22:36:26.669903 13305 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:36:26.679697 13305 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:36:26.680862 13305 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:36:26.702790 13305 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f241ea7630914735a806c4b2935b467d: Generated new TSK 0
I20260925 22:36:26.703557 13305 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.6.246.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.246.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.246.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/krb5kdc/oryx_127.6.246.193.keytab.
Entry for principal oryx/127.6.246.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/krb5kdc/oryx_127.6.246.193.keytab.
I20260925 22:36:26.737521 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.193:0
--local_ip_for_outbound_sockets=127.6.246.193
--webserver_interface=127.6.246.193
--webserver_port=0
--tserver_master_addrs=127.6.246.254:44671
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/krb5kdc/oryx_127.6.246.193.keytab
--principal=oryx/127.6.246.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:39015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:36:26.846354 13317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:36:26.846638 13317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:36:26.846716 13317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:36:26.850652 13317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:36:26.850733 13317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:36:26.850818 13317 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.193
I20260925 22:36:26.855868 13317 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:39015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/krb5kdc/oryx_127.6.246.193.keytab
--principal=oryx/127.6.246.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.246.193
--webserver_port=0
--tserver_master_addrs=127.6.246.254:44671
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13317
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.193
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:36:26.856868 13317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:36:26.857836 13317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:36:26.867182 13325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:26.867234 13322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:26.867216 13323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:26.867364 13317 server_base.cc:1034] running on GCE node
I20260925 22:36:26.867830 13317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:36:26.868410 13317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:36:26.869606 13317 hybrid_clock.cc:648] HybridClock initialized: now 1790375786869576 us; error 47 us; skew 500 ppm
Sep 25 22:36:26 dist-test-slave-wjdr krb5kdc[13210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375786, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.246.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:36:26.872285 13317 init.cc:377] Logged in from keytab as oryx/127.6.246.193@KRBTEST.COM (short username oryx)
I20260925 22:36:26.873247 13317 webserver.cc:460] Webserver started at http://127.6.246.193:38061/ using document root <none> and password file <none>
I20260925 22:36:26.873816 13317 fs_manager.cc:359] Metadata directory not provided
I20260925 22:36:26.873932 13317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:36:26.874156 13317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:36:26.876143 13317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-0/data/instance:
uuid: "67b78fd0bb194aa0b4b9cefa890e343f"
format_stamp: "Formatted at 2026-09-25 22:36:26 on dist-test-slave-wjdr"
server_key: "edd0397cf1c9470b14bce394471c93d2"
server_key_iv: "a5d16793b5211f9a9b18961053e8473a"
server_key_version: "encryptionkey@0"
I20260925 22:36:26.876730 13317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-0/wal/instance:
uuid: "67b78fd0bb194aa0b4b9cefa890e343f"
format_stamp: "Formatted at 2026-09-25 22:36:26 on dist-test-slave-wjdr"
server_key: "edd0397cf1c9470b14bce394471c93d2"
server_key_iv: "a5d16793b5211f9a9b18961053e8473a"
server_key_version: "encryptionkey@0"
I20260925 22:36:26.880357 13317 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260925 22:36:26.882965 13332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:26.884186 13317 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:36:26.884326 13317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-0/wal
uuid: "67b78fd0bb194aa0b4b9cefa890e343f"
format_stamp: "Formatted at 2026-09-25 22:36:26 on dist-test-slave-wjdr"
server_key: "edd0397cf1c9470b14bce394471c93d2"
server_key_iv: "a5d16793b5211f9a9b18961053e8473a"
server_key_version: "encryptionkey@0"
I20260925 22:36:26.884438 13317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:36:26.899331 13317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:36:26.902163 13317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:36:26.902361 13317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:36:26.902930 13317 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:36:26.903700 13317 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:36:26.903767 13317 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:26.903836 13317 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:36:26.903882 13317 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:26.912510 13317 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.193:42341
I20260925 22:36:26.912561 13445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.193:42341 every 8 connection(s)
I20260925 22:36:26.913357 13317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
Sep 25 22:36:26 dist-test-slave-wjdr krb5kdc[13210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375786, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.246.193@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM
I20260925 22:36:26.923568 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 13317
I20260925 22:36:26.923698 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-0/wal/instance
I20260925 22:36:26.924000 7131 external_mini_cluster.cc:1428] Setting key c7fa1356dbe36d213e96c9be6d36b9f8
I20260925 22:36:26.926622 13294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:26.915086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.193:41711 (local address 127.6.246.254:44671)
0925 22:36:26.915216 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:26.915220 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:26.916481 (+ 1261us) server_negotiation.cc:407] Connection header received
0925 22:36:26.917339 (+ 858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:26.917341 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:26.917399 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:26.917467 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:26.918572 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:26.919108 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:26.920145 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:26.920363 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:26.920443 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:26.922766 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:26.922769 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:26.922771 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:26.922792 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:26.924349 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:26.924917 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:26.924920 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:26.924922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:26.925009 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:26.925415 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:26.925419 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:26.925421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:26.925642 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:26.925745 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:26.926238 (+ 493us) server_negotiation.cc:299] Negotiation successful
0925 22:36:26.926380 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 22:36:26.927220 13448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:26.915436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:44671 (local address 127.6.246.193:41711)
0925 22:36:26.916310 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:26.916358 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:26.917142 (+ 784us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:26.917612 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:26.917618 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:26.917987 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:26.918453 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:26.918465 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:26.919209 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:26.919211 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:26.920050 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:26.920059 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:26.920533 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:26.920537 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:26.920683 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:26.921250 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:26.921266 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:26.922626 (+ 1360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:36:26.924489 (+ 1863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:26.924495 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:26.924510 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:26.924804 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:26.925117 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:26.925120 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:26.925121 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:26.925305 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:26.925748 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:36:26.925753 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:36:26.925987 (+ 234us) client_negotiation.cc:769] Sending connection context
0925 22:36:26.926136 (+ 149us) client_negotiation.cc:240] Negotiation successful
0925 22:36:26.926393 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":701,"thread_start_us":568,"threads_started":1}
I20260925 22:36:26.928500 13446 heartbeater.cc:347] Connected to a master server at 127.6.246.254:44671
I20260925 22:36:26.928715 13446 heartbeater.cc:464] Registering TS with master...
I20260925 22:36:26.929168 13446 heartbeater.cc:511] Master 127.6.246.254:44671 requested a full tablet report, sending...
I20260925 22:36:26.930580 13255 ts_manager.cc:194] Registered new tserver with Master: 67b78fd0bb194aa0b4b9cefa890e343f (127.6.246.193:42341)
I20260925 22:36:26.931711 13255 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.246.193@KRBTEST.COM'} at 127.6.246.193:41711
WARNING: no policy specified for oryx/127.6.246.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.246.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.246.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/krb5kdc/oryx_127.6.246.194.keytab.
Entry for principal oryx/127.6.246.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/krb5kdc/oryx_127.6.246.194.keytab.
I20260925 22:36:26.951071 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.194:0
--local_ip_for_outbound_sockets=127.6.246.194
--webserver_interface=127.6.246.194
--webserver_port=0
--tserver_master_addrs=127.6.246.254:44671
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/krb5kdc/oryx_127.6.246.194.keytab
--principal=oryx/127.6.246.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:39015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:36:27.058382 13451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:36:27.058657 13451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:36:27.058723 13451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:36:27.062870 13451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:36:27.062924 13451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:36:27.063012 13451 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.194
I20260925 22:36:27.068084 13451 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:39015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/krb5kdc/oryx_127.6.246.194.keytab
--principal=oryx/127.6.246.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.246.194
--webserver_port=0
--tserver_master_addrs=127.6.246.254:44671
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13451
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.194
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:36:27.069080 13451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:36:27.070030 13451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:36:27.079164 13456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:27.079319 13451 server_base.cc:1034] running on GCE node
W20260925 22:36:27.079169 13459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:27.079164 13457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:27.079814 13451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:36:27.080422 13451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:36:27.081615 13451 hybrid_clock.cc:648] HybridClock initialized: now 1790375787081585 us; error 47 us; skew 500 ppm
Sep 25 22:36:27 dist-test-slave-wjdr krb5kdc[13210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375787, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.246.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:36:27.084197 13451 init.cc:377] Logged in from keytab as oryx/127.6.246.194@KRBTEST.COM (short username oryx)
I20260925 22:36:27.085040 13451 webserver.cc:460] Webserver started at http://127.6.246.194:45597/ using document root <none> and password file <none>
I20260925 22:36:27.085548 13451 fs_manager.cc:359] Metadata directory not provided
I20260925 22:36:27.085605 13451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:36:27.085830 13451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:36:27.087793 13451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-1/data/instance:
uuid: "8d3bf93a4c6844cf9013e229796c55d1"
format_stamp: "Formatted at 2026-09-25 22:36:27 on dist-test-slave-wjdr"
server_key: "d228e1a0fcfd9ca877cc88d3c0cf0b96"
server_key_iv: "df321dcab389123d026b57aa93b6a364"
server_key_version: "encryptionkey@0"
I20260925 22:36:27.088398 13451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-1/wal/instance:
uuid: "8d3bf93a4c6844cf9013e229796c55d1"
format_stamp: "Formatted at 2026-09-25 22:36:27 on dist-test-slave-wjdr"
server_key: "d228e1a0fcfd9ca877cc88d3c0cf0b96"
server_key_iv: "df321dcab389123d026b57aa93b6a364"
server_key_version: "encryptionkey@0"
I20260925 22:36:27.091907 13451 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:36:27.094221 13466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:27.095429 13451 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:36:27.095532 13451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-1/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-1/wal
uuid: "8d3bf93a4c6844cf9013e229796c55d1"
format_stamp: "Formatted at 2026-09-25 22:36:27 on dist-test-slave-wjdr"
server_key: "d228e1a0fcfd9ca877cc88d3c0cf0b96"
server_key_iv: "df321dcab389123d026b57aa93b6a364"
server_key_version: "encryptionkey@0"
I20260925 22:36:27.095633 13451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:36:27.109225 13451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:36:27.112036 13451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:36:27.112222 13451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:36:27.112843 13451 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:36:27.113627 13451 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:36:27.113672 13451 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:27.113737 13451 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:36:27.113797 13451 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:27.122290 13451 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.194:36135
I20260925 22:36:27.122313 13579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.194:36135 every 8 connection(s)
I20260925 22:36:27.123142 13451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
I20260925 22:36:27.126839 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 13451
I20260925 22:36:27.126938 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-1/wal/instance
I20260925 22:36:27.127238 7131 external_mini_cluster.cc:1428] Setting key f802cb8ad6d7b6825de6a2f9eae521bc
Sep 25 22:36:27 dist-test-slave-wjdr krb5kdc[13210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375787, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.246.194@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM
WARNING: no policy specified for oryx/127.6.246.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.246.195@KRBTEST.COM" created.
I20260925 22:36:27.140172 13294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:27.124915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.194:38551 (local address 127.6.246.254:44671)
0925 22:36:27.125055 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:27.125059 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:27.125817 (+ 758us) server_negotiation.cc:407] Connection header received
0925 22:36:27.127171 (+ 1354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:27.127175 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:27.127233 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:27.127334 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:27.128488 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:27.129006 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:27.130102 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:27.130322 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:27.130435 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:27.136357 (+ 5922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:27.136361 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:27.136363 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:27.136387 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:27.137961 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:27.138574 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:27.138581 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:27.138583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:27.138678 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:27.139006 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:27.139013 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:27.139018 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:27.139282 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:27.139393 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:27.139857 (+ 464us) server_negotiation.cc:299] Negotiation successful
0925 22:36:27.139988 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 22:36:27.141014 13582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:27.125266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:44671 (local address 127.6.246.194:38551)
0925 22:36:27.125674 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:27.125713 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:27.126933 (+ 1220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:27.127503 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:27.127510 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:27.127845 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:27.128309 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:27.128321 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:27.129140 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:27.129144 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:27.129981 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:27.129990 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:27.130468 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:27.130472 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:27.130638 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:27.134723 (+ 4085us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:27.134739 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:27.136183 (+ 1444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:36:27.138056 (+ 1873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:27.138061 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:27.138072 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:27.138461 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:27.138772 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:27.138775 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:27.138776 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:27.138910 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:27.139393 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:36:27.139398 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:36:27.139620 (+ 222us) client_negotiation.cc:769] Sending connection context
0925 22:36:27.139809 (+ 189us) client_negotiation.cc:240] Negotiation successful
0925 22:36:27.140124 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":111,"threads_started":1}
I20260925 22:36:27.142175 13580 heartbeater.cc:347] Connected to a master server at 127.6.246.254:44671
I20260925 22:36:27.142416 13580 heartbeater.cc:464] Registering TS with master...
I20260925 22:36:27.142871 13580 heartbeater.cc:511] Master 127.6.246.254:44671 requested a full tablet report, sending...
I20260925 22:36:27.143894 13255 ts_manager.cc:194] Registered new tserver with Master: 8d3bf93a4c6844cf9013e229796c55d1 (127.6.246.194:36135)
I20260925 22:36:27.144747 13255 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.246.194@KRBTEST.COM'} at 127.6.246.194:38551
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.246.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/krb5kdc/oryx_127.6.246.195.keytab.
Entry for principal oryx/127.6.246.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/krb5kdc/oryx_127.6.246.195.keytab.
I20260925 22:36:27.153101 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.195:0
--local_ip_for_outbound_sockets=127.6.246.195
--webserver_interface=127.6.246.195
--webserver_port=0
--tserver_master_addrs=127.6.246.254:44671
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/krb5kdc/oryx_127.6.246.195.keytab
--principal=oryx/127.6.246.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:39015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:36:27.259531 13585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:36:27.259801 13585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:36:27.259881 13585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:36:27.263867 13585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:36:27.263944 13585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:36:27.264025 13585 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.195
I20260925 22:36:27.269047 13585 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:39015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/krb5kdc/oryx_127.6.246.195.keytab
--principal=oryx/127.6.246.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.246.195
--webserver_port=0
--tserver_master_addrs=127.6.246.254:44671
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13585
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.195
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:36:27.270001 13585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:36:27.270925 13585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:36:27.279958 13593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:27.279995 13590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:27.280005 13591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:27.280153 13585 server_base.cc:1034] running on GCE node
I20260925 22:36:27.280594 13585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:36:27.281209 13585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:36:27.282395 13585 hybrid_clock.cc:648] HybridClock initialized: now 1790375787282386 us; error 29 us; skew 500 ppm
Sep 25 22:36:27 dist-test-slave-wjdr krb5kdc[13210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375787, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.246.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:36:27.285022 13585 init.cc:377] Logged in from keytab as oryx/127.6.246.195@KRBTEST.COM (short username oryx)
I20260925 22:36:27.285806 13585 webserver.cc:460] Webserver started at http://127.6.246.195:38517/ using document root <none> and password file <none>
I20260925 22:36:27.286355 13585 fs_manager.cc:359] Metadata directory not provided
I20260925 22:36:27.286453 13585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:36:27.286665 13585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:36:27.288730 13585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-2/data/instance:
uuid: "d941f16284f0429381d37c34d0eaae2a"
format_stamp: "Formatted at 2026-09-25 22:36:27 on dist-test-slave-wjdr"
server_key: "4c850abc83584ab2d418603cac494190"
server_key_iv: "6b75aa2e350985d879c0209a33759e4c"
server_key_version: "encryptionkey@0"
I20260925 22:36:27.289289 13585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-2/wal/instance:
uuid: "d941f16284f0429381d37c34d0eaae2a"
format_stamp: "Formatted at 2026-09-25 22:36:27 on dist-test-slave-wjdr"
server_key: "4c850abc83584ab2d418603cac494190"
server_key_iv: "6b75aa2e350985d879c0209a33759e4c"
server_key_version: "encryptionkey@0"
I20260925 22:36:27.292797 13585 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260925 22:36:27.295035 13600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:27.296221 13585 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:36:27.296372 13585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-2/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-2/wal
uuid: "d941f16284f0429381d37c34d0eaae2a"
format_stamp: "Formatted at 2026-09-25 22:36:27 on dist-test-slave-wjdr"
server_key: "4c850abc83584ab2d418603cac494190"
server_key_iv: "6b75aa2e350985d879c0209a33759e4c"
server_key_version: "encryptionkey@0"
I20260925 22:36:27.296494 13585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:36:27.314002 13585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:36:27.316735 13585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:36:27.316924 13585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:36:27.317466 13585 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:36:27.318282 13585 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:36:27.318346 13585 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:27.318411 13585 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:36:27.318459 13585 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:27.327083 13585 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.195:44879
I20260925 22:36:27.327106 13713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.195:44879 every 8 connection(s)
I20260925 22:36:27.327930 13585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
I20260925 22:36:27.328394 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 13585
I20260925 22:36:27.328624 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-2/wal/instance
I20260925 22:36:27.328912 7131 external_mini_cluster.cc:1428] Setting key 66af2096a9726098fe324a1686636bba
Sep 25 22:36:27 dist-test-slave-wjdr krb5kdc[13210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375787, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.246.195@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM
I20260925 22:36:27.341179 13294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:27.329843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.195:54123 (local address 127.6.246.254:44671)
0925 22:36:27.329974 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:27.329983 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:27.330904 (+ 921us) server_negotiation.cc:407] Connection header received
0925 22:36:27.332237 (+ 1333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:27.332240 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:27.332292 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:27.332380 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:27.333496 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:27.334049 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:27.335147 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:27.335377 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:27.335483 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:27.337928 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:27.337932 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:27.337935 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:27.337981 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:27.339186 (+ 1205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:27.339655 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:27.339660 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:27.339662 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:27.339745 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:27.340074 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:27.340078 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:27.340080 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:27.340255 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:27.340366 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:27.340896 (+ 530us) server_negotiation.cc:299] Negotiation successful
0925 22:36:27.341014 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 22:36:27.341959 13716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:27.330290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:44671 (local address 127.6.246.195:54123)
0925 22:36:27.330758 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:27.330808 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:27.332041 (+ 1233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:27.332554 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:27.332561 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:27.332909 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:27.333364 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:27.333376 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:27.334166 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:27.334169 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:27.335025 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:27.335034 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:27.335514 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:27.335518 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:27.335682 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:27.336254 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:27.336270 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:27.337691 (+ 1421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:36:27.339297 (+ 1606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:27.339302 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:27.339316 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:27.339555 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:27.339837 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:27.339840 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:27.339842 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:27.339980 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:27.340364 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:36:27.340371 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:36:27.340655 (+ 284us) client_negotiation.cc:769] Sending connection context
0925 22:36:27.340850 (+ 195us) client_negotiation.cc:240] Negotiation successful
0925 22:36:27.341139 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":139,"threads_started":1}
I20260925 22:36:27.343070 13714 heartbeater.cc:347] Connected to a master server at 127.6.246.254:44671
I20260925 22:36:27.343266 13714 heartbeater.cc:464] Registering TS with master...
I20260925 22:36:27.343667 13714 heartbeater.cc:511] Master 127.6.246.254:44671 requested a full tablet report, sending...
I20260925 22:36:27.344583 13255 ts_manager.cc:194] Registered new tserver with Master: d941f16284f0429381d37c34d0eaae2a (127.6.246.195:44879)
I20260925 22:36:27.345116 13255 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.246.195@KRBTEST.COM'} at 127.6.246.195:54123
I20260925 22:36:27.354586 7131 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.6.246.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.246.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.246.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/krb5kdc/oryx_127.6.246.253.keytab.
Entry for principal oryx/127.6.246.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/krb5kdc/oryx_127.6.246.253.keytab.
I20260925 22:36:27.379618 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.253:40799
--webserver_interface=127.6.246.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/krb5kdc/oryx_127.6.246.253.keytab
--principal=oryx/127.6.246.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:39015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.246.254:44671,127.6.246.253:40799
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:36:27.499619 13719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:36:27.499858 13719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:36:27.499933 13719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:36:27.503911 13719 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:36:27.503962 13719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:36:27.503980 13719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:36:27.503997 13719 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:36:27.504011 13719 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:36:27.509207 13719 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:39015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.6.246.254:44671,127.6.246.253:40799
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/krb5kdc/oryx_127.6.246.253.keytab
--principal=oryx/127.6.246.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.253:40799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.246.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
W20260925 22:36:27.509359 13719 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.6.246.254:44671,127.6.246.253:40799'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260925 22:36:27.510092 13719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:36:27.510957 13719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:36:27.518934 13725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:27.518947 13727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:27.518977 13724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:27.518992 13719 server_base.cc:1034] running on GCE node
I20260925 22:36:27.519673 13719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:36:27.520674 13719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:36:27.521879 13719 hybrid_clock.cc:648] HybridClock initialized: now 1790375787521852 us; error 40 us; skew 500 ppm
Sep 25 22:36:27 dist-test-slave-wjdr krb5kdc[13210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375787, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.246.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:36:27.524566 13719 init.cc:377] Logged in from keytab as oryx/127.6.246.253@KRBTEST.COM (short username oryx)
I20260925 22:36:27.525306 13719 webserver.cc:460] Webserver started at http://127.6.246.253:44503/ using document root <none> and password file <none>
I20260925 22:36:27.525815 13719 fs_manager.cc:359] Metadata directory not provided
I20260925 22:36:27.525872 13719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:36:27.526065 13719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:36:27.527933 13719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/master-1/data/instance:
uuid: "d481871900894bbdbed9d74795df7ece"
format_stamp: "Formatted at 2026-09-25 22:36:27 on dist-test-slave-wjdr"
server_key: "bfbc63cd2f7e392f218798b259b3b5ac"
server_key_iv: "525eb75ddc5d6c0b2521f961c5ab65cd"
server_key_version: "encryptionkey@0"
I20260925 22:36:27.528519 13719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/master-1/wal/instance:
uuid: "d481871900894bbdbed9d74795df7ece"
format_stamp: "Formatted at 2026-09-25 22:36:27 on dist-test-slave-wjdr"
server_key: "bfbc63cd2f7e392f218798b259b3b5ac"
server_key_iv: "525eb75ddc5d6c0b2521f961c5ab65cd"
server_key_version: "encryptionkey@0"
I20260925 22:36:27.531837 13719 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:36:27.534075 13734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:27.535161 13719 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 22:36:27.535259 13719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/master-1/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/master-1/wal
uuid: "d481871900894bbdbed9d74795df7ece"
format_stamp: "Formatted at 2026-09-25 22:36:27 on dist-test-slave-wjdr"
server_key: "bfbc63cd2f7e392f218798b259b3b5ac"
server_key_iv: "525eb75ddc5d6c0b2521f961c5ab65cd"
server_key_version: "encryptionkey@0"
I20260925 22:36:27.535378 13719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:36:27.554351 13719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:36:27.557173 13719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:36:27.557364 13719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:36:27.564565 13719 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.253:40799
I20260925 22:36:27.564851 13786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.253:40799 every 8 connection(s)
I20260925 22:36:27.565474 13719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/master-1/data/info.pb
I20260925 22:36:27.566061 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 13719
I20260925 22:36:27.568225 13787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:36:27.572396 13787 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.246.254" port: 44671 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 22:36:27 dist-test-slave-wjdr krb5kdc[13210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375786, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.246.253@KRBTEST.COM
Sep 25 22:36:27 dist-test-slave-wjdr krb5kdc[13210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375787, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.246.253@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM
I20260925 22:36:27.581226 13790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:27.567440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57924 (local address 127.6.246.253:40799)
0925 22:36:27.567807 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:27.567827 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:27.567869 (+ 42us) server_negotiation.cc:407] Connection header received
0925 22:36:27.568630 (+ 761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:27.568648 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:27.568962 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:27.569232 (+ 270us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:27.570132 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:27.571131 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:27.572277 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:27.572997 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:27.573133 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:27.576103 (+ 2970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:27.576109 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:27.576120 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:27.576151 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:27.578347 (+ 2196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:27.578760 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:27.578765 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:27.578771 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:27.578887 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:27.579155 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:27.579160 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:27.579161 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:27.579554 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:27.579781 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:27.580060 (+ 279us) server_negotiation.cc:299] Negotiation successful
0925 22:36:27.580354 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":123,"threads_started":1}
I20260925 22:36:27.584249 13294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:27.574365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53052 (local address 127.6.246.254:44671)
0925 22:36:27.574519 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:27.574524 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:27.574854 (+ 330us) server_negotiation.cc:407] Connection header received
0925 22:36:27.575010 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:27.575012 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:27.575056 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:27.575132 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:27.575982 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:27.576867 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:27.578276 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:27.578516 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:27.578650 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:27.580818 (+ 2168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:27.580830 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:27.580833 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:27.580862 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:27.582498 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:27.582951 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:27.582956 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:27.582958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:27.583036 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:27.583300 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:27.583305 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:27.583307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:27.583578 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:27.583682 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:27.583977 (+ 295us) server_negotiation.cc:299] Negotiation successful
0925 22:36:27.584090 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 22:36:27.584276 13791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:27.574300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:44671 (local address 127.0.0.1:53052)
0925 22:36:27.574711 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:27.574765 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:27.574912 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:27.575206 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:27.575211 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:27.575645 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:27.575850 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:27.575860 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:27.577026 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:27.577031 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:27.578141 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:27.578153 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:27.578648 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:27.578652 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:27.578776 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:27.579282 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:27.579293 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:27.580686 (+ 1393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:36:27.582603 (+ 1917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:27.582612 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:27.582617 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:27.582833 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:27.583126 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:27.583130 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:27.583132 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:27.583212 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:27.583678 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:36:27.583684 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:36:27.583792 (+ 108us) client_negotiation.cc:769] Sending connection context
0925 22:36:27.583922 (+ 130us) client_negotiation.cc:240] Negotiation successful
0925 22:36:27.584090 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":150,"threads_started":1}
I20260925 22:36:27.585472 13787 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.246.253" port: 40799 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 22:36:27 dist-test-slave-wjdr krb5kdc[13210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375787, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.246.253@KRBTEST.COM for oryx/127.6.246.253@KRBTEST.COM
I20260925 22:36:27.593189 13791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:27.585932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.253:40799 (local address 127.0.0.1:57938)
0925 22:36:27.586077 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:27.586088 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:27.586187 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:27.586388 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:27.586390 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:27.586512 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:27.586595 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:27.586600 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:27.587748 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:27.587751 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:27.588316 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:27.588324 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:27.588795 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:27.588798 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:27.588871 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:27.589297 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:27.589308 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:27.590489 (+ 1181us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:36:27.591924 (+ 1435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:27.591927 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:27.591928 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:27.592102 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:27.592352 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:27.592355 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:27.592356 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:27.592434 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:27.592811 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:36:27.592814 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:36:27.592884 (+ 70us) client_negotiation.cc:769] Sending connection context
0925 22:36:27.592963 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 22:36:27.593051 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113}
I20260925 22:36:27.593240 13790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:27.585938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57938 (local address 127.6.246.253:40799)
0925 22:36:27.586033 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:27.586036 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:27.586138 (+ 102us) server_negotiation.cc:407] Connection header received
0925 22:36:27.586265 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:27.586267 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:27.586301 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:27.586362 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:27.586678 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:27.587624 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:27.588436 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:27.588691 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:27.588764 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:27.590592 (+ 1828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:27.590595 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:27.590597 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:27.590616 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:27.591802 (+ 1186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:27.592177 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:27.592180 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:27.592181 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:27.592249 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:27.592572 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:27.592574 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:27.592575 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:27.592722 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:27.592795 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:27.593005 (+ 210us) server_negotiation.cc:299] Negotiation successful
0925 22:36:27.593099 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":8,"server-negotiator.queue_time_us":38}
I20260925 22:36:27.595912 13787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece: Bootstrap starting.
I20260925 22:36:27.598163 13787 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece: Neither blocks nor log segments found. Creating new log.
I20260925 22:36:27.598850 13787 log.cc:824] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece: Log is configured to *not* fsync() on all Append() calls
I20260925 22:36:27.600751 13787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece: No bootstrap required, opened a new log
I20260925 22:36:27.603566 13787 raft_consensus.cc:348] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f241ea7630914735a806c4b2935b467d" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 44671 } } peers { permanent_uuid: "d481871900894bbdbed9d74795df7ece" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 40799 } }
I20260925 22:36:27.603744 13787 raft_consensus.cc:374] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:36:27.603806 13787 raft_consensus.cc:729] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d481871900894bbdbed9d74795df7ece, State: Initialized, Role: FOLLOWER
I20260925 22:36:27.604207 13787 consensus_queue.cc:260] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f241ea7630914735a806c4b2935b467d" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 44671 } } peers { permanent_uuid: "d481871900894bbdbed9d74795df7ece" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 40799 } }
I20260925 22:36:27.604686 13793 sys_catalog.cc:455] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f241ea7630914735a806c4b2935b467d" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 44671 } } peers { permanent_uuid: "d481871900894bbdbed9d74795df7ece" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 40799 } } }
I20260925 22:36:27.604799 13793 sys_catalog.cc:458] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:36:27.605378 13787 sys_catalog.cc:564] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece [sys.catalog]: configured and running, proceeding with master startup.
W20260925 22:36:27.609711 13804 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:36:27.609843 13804 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:36:27.611326 13719 master_runner.cc:428] Detected that this master d481871900894bbdbed9d74795df7ece is joining an existing cluster
I20260925 22:36:27.611382 13719 master_runner.cc:432] Initiating AddMaster RPC to add 127.6.246.253:40799
I20260925 22:36:27.621600 13790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:27.613812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57952 (local address 127.6.246.253:40799)
0925 22:36:27.613912 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:27.613917 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:27.614208 (+ 291us) server_negotiation.cc:407] Connection header received
0925 22:36:27.614328 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:27.614334 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:27.614383 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:27.614484 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:27.614952 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:27.615683 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:27.616908 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:27.617229 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:27.617316 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:27.618456 (+ 1140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:27.618460 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:27.618463 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:27.618483 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:27.620078 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:27.620495 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:27.620498 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:27.620500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:27.620575 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:27.620834 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:27.620837 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:27.620839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:27.620991 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:27.621089 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:27.621323 (+ 234us) server_negotiation.cc:299] Negotiation successful
0925 22:36:27.621460 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":77,"server-negotiator.queue_time_us":32}
I20260925 22:36:27.621765 13810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:27.613643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.253:40799 (local address 127.0.0.1:57952)
0925 22:36:27.614106 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:27.614119 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:27.614236 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:27.614496 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:27.614499 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:27.614742 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:27.614846 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:27.614851 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:27.615806 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:27.615809 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:27.616797 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:27.616808 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:27.617318 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:27.617321 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:27.617445 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:27.617936 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:27.617949 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:27.618329 (+ 380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:36:27.620198 (+ 1869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:27.620202 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:27.620205 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:27.620391 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:27.620660 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:27.620665 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:27.620667 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:27.620746 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:27.621078 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:36:27.621080 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:36:27.621172 (+ 92us) client_negotiation.cc:769] Sending connection context
0925 22:36:27.621277 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 22:36:27.621412 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"mutex_wait_us":48,"thread_start_us":239,"threads_started":1}
I20260925 22:36:27.622109 13294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:27.613940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53062 (local address 127.6.246.254:44671)
0925 22:36:27.614062 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:27.614066 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:27.614145 (+ 79us) server_negotiation.cc:407] Connection header received
0925 22:36:27.614406 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:27.614409 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:27.614458 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:27.614535 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:27.615156 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:27.615886 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:27.616744 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:27.617039 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:27.617137 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:27.618433 (+ 1296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:27.618437 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:27.618440 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:27.618465 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:27.620322 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:27.620814 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:27.620817 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:27.620819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:27.620899 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:27.621220 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:27.621223 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:27.621225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:27.621421 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:27.621489 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:27.621834 (+ 345us) server_negotiation.cc:299] Negotiation successful
0925 22:36:27.621947 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 22:36:27.622582 13811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:27.613643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:44671 (local address 127.0.0.1:53062)
0925 22:36:27.614064 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:27.614079 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:27.614256 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:27.614647 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:27.614650 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:27.614901 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:27.615034 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:27.615042 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:27.615985 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:27.615990 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:27.616633 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:27.616641 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:27.617165 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:27.617169 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:27.617277 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:27.617781 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:27.617797 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:27.618280 (+ 483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:36:27.620442 (+ 2162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:27.620445 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:27.620447 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:27.620700 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:27.621030 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:27.621033 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:27.621036 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:27.621116 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:27.621562 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:36:27.621566 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:36:27.621652 (+ 86us) client_negotiation.cc:769] Sending connection context
0925 22:36:27.622287 (+ 635us) client_negotiation.cc:240] Negotiation successful
0925 22:36:27.622418 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"spinlock_wait_cycles":27904,"thread_start_us":121,"threads_started":1}
Sep 25 22:36:27 dist-test-slave-wjdr krb5kdc[13210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375786, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.246.254@KRBTEST.COM for oryx/127.6.246.253@KRBTEST.COM
I20260925 22:36:27.635231 13812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:27.626319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.253:40799 (local address 127.0.0.1:57956)
0925 22:36:27.626627 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:27.626665 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:27.626818 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:27.627097 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:27.627103 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:27.627446 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:27.627539 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:27.627545 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:27.628240 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:27.628243 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:27.628977 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:27.628986 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:27.629419 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:27.629422 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:27.629535 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:27.630030 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:27.630041 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:27.631552 (+ 1511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:36:27.633278 (+ 1726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:27.633288 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:27.633295 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:27.633547 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:27.634058 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:27.634061 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:27.634063 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:27.634155 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:27.634568 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:36:27.634575 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:36:27.634699 (+ 124us) client_negotiation.cc:769] Sending connection context
0925 22:36:27.634846 (+ 147us) client_negotiation.cc:240] Negotiation successful
0925 22:36:27.635060 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":87,"threads_started":1}
I20260925 22:36:27.635251 13790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:27.626356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57956 (local address 127.6.246.253:40799)
0925 22:36:27.626453 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:27.626456 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:27.626740 (+ 284us) server_negotiation.cc:407] Connection header received
0925 22:36:27.626935 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:27.626938 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:27.626988 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:27.627065 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:27.627648 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:27.628127 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:27.629093 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:27.629315 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:27.629418 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:27.631690 (+ 2272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:27.631694 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:27.631697 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:27.631719 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:27.633146 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:27.633663 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:27.633666 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:27.633668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:27.633777 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:27.634255 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:27.634258 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:27.634261 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:27.634457 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:27.634554 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:27.634913 (+ 359us) server_negotiation.cc:299] Negotiation successful
0925 22:36:27.635031 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 22:36:27.637490 13255 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.6.246.253:40799: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "d481871900894bbdbed9d74795df7ece"
member_type: NON_VOTER
last_known_addr {
host: "127.6.246.253"
port: 40799
}
attrs {
promote: true
}
}
dest_uuid: "f241ea7630914735a806c4b2935b467d"
cas_config_opid_index: -1
I20260925 22:36:27.638105 13255 consensus_queue.cc:237] T 00000000000000000000000000000000 P f241ea7630914735a806c4b2935b467d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f241ea7630914735a806c4b2935b467d" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 44671 } } peers { permanent_uuid: "d481871900894bbdbed9d74795df7ece" member_type: NON_VOTER last_known_addr { host: "127.6.246.253" port: 40799 } attrs { promote: true } }
I20260925 22:36:27.639652 13815 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f241ea7630914735a806c4b2935b467d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER d481871900894bbdbed9d74795df7ece (127.6.246.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f241ea7630914735a806c4b2935b467d" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 44671 } } peers { permanent_uuid: "d481871900894bbdbed9d74795df7ece" member_type: NON_VOTER last_known_addr { host: "127.6.246.253" port: 40799 } attrs { promote: true } } }
I20260925 22:36:27.640295 13761 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:36:27.641646 13761 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece [term 1 FOLLOWER]: Refusing update from remote peer f241ea7630914735a806c4b2935b467d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260925 22:36:27.641994 13815 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.6.246.253:40799
I20260925 22:36:27.642330 13815 sys_catalog.cc:455] T 00000000000000000000000000000000 P f241ea7630914735a806c4b2935b467d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f241ea7630914735a806c4b2935b467d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f241ea7630914735a806c4b2935b467d" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 44671 } } peers { permanent_uuid: "d481871900894bbdbed9d74795df7ece" member_type: NON_VOTER last_known_addr { host: "127.6.246.253" port: 40799 } attrs { promote: true } } }
I20260925 22:36:27.642678 13719 master.cc:537] Master@127.6.246.253:40799 shutting down...
I20260925 22:36:27.643033 13815 sys_catalog.cc:458] T 00000000000000000000000000000000 P f241ea7630914735a806c4b2935b467d [sys.catalog]: This master's current role is: LEADER
I20260925 22:36:27.642520 13814 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f241ea7630914735a806c4b2935b467d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d481871900894bbdbed9d74795df7ece" member_type: NON_VOTER last_known_addr { host: "127.6.246.253" port: 40799 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 22:36:27.644729 13242 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260925 22:36:27.646839 13719 raft_consensus.cc:2229] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:36:27.647120 13719 raft_consensus.cc:2258] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:36:27.647238 13719 tablet_replica.cc:330] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece: stopping tablet replica
WARNING: no policy specified for oryx/127.6.246.252@KRBTEST.COM; defaulting to no policy
W20260925 22:36:27.649226 13242 consensus_peers.cc:487] T 00000000000000000000000000000000 P f241ea7630914735a806c4b2935b467d -> Peer d481871900894bbdbed9d74795df7ece (127.6.246.253:40799): Couldn't send request to peer d481871900894bbdbed9d74795df7ece. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.246.252@KRBTEST.COM" created.
I20260925 22:36:27.653710 13719 master.cc:552] Master@127.6.246.253:40799 shutdown complete.
I20260925 22:36:27.653781 13719 master_runner.cc:305] Clearing existing system tablet
I20260925 22:36:27.654030 13719 fs_manager.cc:359] Metadata directory not provided
I20260925 22:36:27.654086 13719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:36:27.655390 13719 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 22:36:27.656301 13821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:27.659566 13719 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:36:27.659780 13719 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260925 22:36:27.659826 13719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/master-1/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/master-1/wal
uuid: "d481871900894bbdbed9d74795df7ece"
format_stamp: "Formatted at 2026-09-25 22:36:27 on dist-test-slave-wjdr"
server_key: "bfbc63cd2f7e392f218798b259b3b5ac"
server_key_iv: "525eb75ddc5d6c0b2521f961c5ab65cd"
server_key_version: "encryptionkey@0"
I20260925 22:36:27.662103 13719 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:36:27.665576 13719 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 22:36:27.665664 13719 log.cc:1196] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece: Deleting WAL directory at /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260925 22:36:27.665890 13719 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece: Deleting consensus metadata
I20260925 22:36:27.666071 13719 master_runner.cc:315] Copying system tablet from 127.6.246.254:44671
I20260925 22:36:27.667212 13719 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece: tablet copy: Beginning tablet copy session from remote peer at address 127.6.246.254:44671
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.246.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/krb5kdc/oryx_127.6.246.252.keytab.
Entry for principal oryx/127.6.246.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/krb5kdc/oryx_127.6.246.252.keytab.
I20260925 22:36:27.668202 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.252:46735
--webserver_interface=127.6.246.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/krb5kdc/oryx_127.6.246.252.keytab
--principal=oryx/127.6.246.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:39015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.246.254:44671,127.6.246.253:40799,127.6.246.252:46735
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:36:27.676936 13294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:27.667811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53068 (local address 127.6.246.254:44671)
0925 22:36:27.667986 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:27.667992 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:27.668141 (+ 149us) server_negotiation.cc:407] Connection header received
0925 22:36:27.668288 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:27.668294 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:27.668352 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:27.668428 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:27.668981 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:27.669540 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:27.670850 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:27.671081 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:27.671139 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:27.673248 (+ 2109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:27.673252 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:27.673255 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:27.673282 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:27.675047 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:27.675582 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:27.675586 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:27.675588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:27.675673 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:27.675987 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:27.675990 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:27.675992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:27.676198 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:27.676289 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:27.676663 (+ 374us) server_negotiation.cc:299] Negotiation successful
0925 22:36:27.676776 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260925 22:36:27.677515 13827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:27.667766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:44671 (local address 127.0.0.1:53068)
0925 22:36:27.668057 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:27.668069 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:27.668188 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:27.668592 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:27.668596 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:27.668786 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:27.668864 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:27.668870 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:27.669688 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:27.669694 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:27.670692 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:27.670703 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:27.671760 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:27.671766 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:27.671886 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:27.672603 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:27.672620 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:27.673102 (+ 482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:36:27.675164 (+ 2062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:27.675171 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:27.675176 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:27.675469 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:27.675768 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:27.675773 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:27.675778 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:27.675886 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:27.676332 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:36:27.676335 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:36:27.676428 (+ 93us) client_negotiation.cc:769] Sending connection context
0925 22:36:27.677092 (+ 664us) client_negotiation.cc:240] Negotiation successful
0925 22:36:27.677234 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":87,"threads_started":1}
I20260925 22:36:27.678157 13275 tablet_copy_service.cc:140] P f241ea7630914735a806c4b2935b467d: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer d481871900894bbdbed9d74795df7ece ({username='oryx', principal='oryx/127.6.246.253@KRBTEST.COM'} at 127.0.0.1:53068)
I20260925 22:36:27.678339 13275 tablet_copy_service.cc:161] P f241ea7630914735a806c4b2935b467d: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer d481871900894bbdbed9d74795df7ece at {username='oryx', principal='oryx/127.6.246.253@KRBTEST.COM'} at 127.0.0.1:53068: session id = d481871900894bbdbed9d74795df7ece-00000000000000000000000000000000
I20260925 22:36:27.680069 13275 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P f241ea7630914735a806c4b2935b467d: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 22:36:27.681654 13719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 22:36:27.683845 13719 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece: tablet copy: Starting download of 0 data blocks...
I20260925 22:36:27.684026 13719 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece: tablet copy: Starting download of 1 WAL segments...
I20260925 22:36:27.685614 13719 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 22:36:27.687355 13275 tablet_copy_service.cc:342] P f241ea7630914735a806c4b2935b467d: Request end of tablet copy session d481871900894bbdbed9d74795df7ece-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.6.246.253@KRBTEST.COM'} at 127.0.0.1:53068
I20260925 22:36:27.687462 13275 tablet_copy_service.cc:433] P f241ea7630914735a806c4b2935b467d: ending tablet copy session d481871900894bbdbed9d74795df7ece-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer d481871900894bbdbed9d74795df7ece
I20260925 22:36:27.694366 13719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:36:27.695856 13829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:27.696018 13830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:27.696302 13832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:27.696851 13719 server_base.cc:1034] running on GCE node
I20260925 22:36:27.697002 13719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:36:27.697283 13719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:36:27.698451 13719 hybrid_clock.cc:648] HybridClock initialized: now 1790375787698437 us; error 28 us; skew 500 ppm
Sep 25 22:36:27 dist-test-slave-wjdr krb5kdc[13210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375787, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.246.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:36:27.700035 13719 init.cc:377] Logged in from keytab as oryx/127.6.246.253@KRBTEST.COM (short username oryx)
I20260925 22:36:27.700515 13719 webserver.cc:460] Webserver started at http://127.6.246.253:39677/ using document root <none> and password file <none>
I20260925 22:36:27.700692 13719 fs_manager.cc:359] Metadata directory not provided
I20260925 22:36:27.700783 13719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:36:27.702028 13719 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 22:36:27.702939 13839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:27.703200 13719 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 22:36:27.703274 13719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/master-1/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/master-1/wal
uuid: "d481871900894bbdbed9d74795df7ece"
format_stamp: "Formatted at 2026-09-25 22:36:27 on dist-test-slave-wjdr"
server_key: "bfbc63cd2f7e392f218798b259b3b5ac"
server_key_iv: "525eb75ddc5d6c0b2521f961c5ab65cd"
server_key_version: "encryptionkey@0"
I20260925 22:36:27.703368 13719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:36:27.743955 13719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:36:27.746558 13719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:36:27.751303 13719 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.253:40799
I20260925 22:36:27.751466 13891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.253:40799 every 8 connection(s)
I20260925 22:36:27.751881 13719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/master-1/data/info.pb
I20260925 22:36:27.753705 13892 sys_catalog.cc:263] Verifying existing consensus state
I20260925 22:36:27.754861 13892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece: Bootstrap starting.
I20260925 22:36:27.765532 13892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 22:36:27.765839 13892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece: Bootstrap complete.
I20260925 22:36:27.766381 13892 raft_consensus.cc:348] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f241ea7630914735a806c4b2935b467d" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 44671 } } peers { permanent_uuid: "d481871900894bbdbed9d74795df7ece" member_type: NON_VOTER last_known_addr { host: "127.6.246.253" port: 40799 } attrs { promote: true } }
I20260925 22:36:27.766507 13892 raft_consensus.cc:729] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d481871900894bbdbed9d74795df7ece, State: Initialized, Role: LEARNER
I20260925 22:36:27.766605 13892 consensus_queue.cc:260] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f241ea7630914735a806c4b2935b467d" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 44671 } } peers { permanent_uuid: "d481871900894bbdbed9d74795df7ece" member_type: NON_VOTER last_known_addr { host: "127.6.246.253" port: 40799 } attrs { promote: true } }
I20260925 22:36:27.767016 13892 sys_catalog.cc:564] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:36:27.766916 13895 sys_catalog.cc:455] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f241ea7630914735a806c4b2935b467d" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 44671 } } peers { permanent_uuid: "d481871900894bbdbed9d74795df7ece" member_type: NON_VOTER last_known_addr { host: "127.6.246.253" port: 40799 } attrs { promote: true } } }
I20260925 22:36:27.767215 13895 sys_catalog.cc:458] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece [sys.catalog]: This master's current role is: LEARNER
I20260925 22:36:27.769847 13906 catalog_manager.cc:1259] Loaded cluster ID: 2f47f719071c41b3ad8263c885d14031
I20260925 22:36:27.769949 13906 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece: loading cluster ID for follower catalog manager: success
I20260925 22:36:27.771519 13906 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece: acquiring CA information for follower catalog manager: success
I20260925 22:36:27.772641 13906 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 22:36:27.809019 13828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:36:27.809270 13828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:36:27.809309 13828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:36:27.813230 13828 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:36:27.813280 13828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:36:27.813299 13828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:36:27.813316 13828 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:36:27.813331 13828 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:36:27.818339 13828 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:39015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.6.246.254:44671,127.6.246.253:40799,127.6.246.252:46735
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/krb5kdc/oryx_127.6.246.252.keytab
--principal=oryx/127.6.246.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.252:46735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.246.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:36:27.819281 13828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:36:27.820216 13828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:36:27.828352 13914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:27.828408 13911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:27.828408 13912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:27.828608 13828 server_base.cc:1034] running on GCE node
I20260925 22:36:27.829105 13828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:36:27.830118 13828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:36:27.831285 13828 hybrid_clock.cc:648] HybridClock initialized: now 1790375787831260 us; error 42 us; skew 500 ppm
Sep 25 22:36:27 dist-test-slave-wjdr krb5kdc[13210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375787, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.246.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:36:27.833979 13828 init.cc:377] Logged in from keytab as oryx/127.6.246.252@KRBTEST.COM (short username oryx)
I20260925 22:36:27.834755 13828 webserver.cc:460] Webserver started at http://127.6.246.252:38071/ using document root <none> and password file <none>
I20260925 22:36:27.835269 13828 fs_manager.cc:359] Metadata directory not provided
I20260925 22:36:27.835351 13828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:36:27.835561 13828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:36:27.837512 13828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/master-2/data/instance:
uuid: "3bf1e94e512547acba5b020a5eab47f5"
format_stamp: "Formatted at 2026-09-25 22:36:27 on dist-test-slave-wjdr"
server_key: "3f255fd19ca85953ed35ba2f4f71ed25"
server_key_iv: "829da5a489e5e50393bda36e9c4c7787"
server_key_version: "encryptionkey@0"
I20260925 22:36:27.838039 13828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/master-2/wal/instance:
uuid: "3bf1e94e512547acba5b020a5eab47f5"
format_stamp: "Formatted at 2026-09-25 22:36:27 on dist-test-slave-wjdr"
server_key: "3f255fd19ca85953ed35ba2f4f71ed25"
server_key_iv: "829da5a489e5e50393bda36e9c4c7787"
server_key_version: "encryptionkey@0"
I20260925 22:36:27.841434 13828 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 22:36:27.843629 13921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:27.844938 13828 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:36:27.845072 13828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/master-2/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/master-2/wal
uuid: "3bf1e94e512547acba5b020a5eab47f5"
format_stamp: "Formatted at 2026-09-25 22:36:27 on dist-test-slave-wjdr"
server_key: "3f255fd19ca85953ed35ba2f4f71ed25"
server_key_iv: "829da5a489e5e50393bda36e9c4c7787"
server_key_version: "encryptionkey@0"
I20260925 22:36:27.845178 13828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:36:27.855123 13828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:36:27.857923 13828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:36:27.858103 13828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:36:27.864701 13828 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.252:46735
I20260925 22:36:27.864987 13973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.252:46735 every 8 connection(s)
I20260925 22:36:27.865675 13828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/master-2/data/info.pb
I20260925 22:36:27.866979 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 13828
I20260925 22:36:27.868748 13974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:36:27.873172 13974 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.246.254" port: 44671 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 22:36:27 dist-test-slave-wjdr krb5kdc[13210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375786, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.246.252@KRBTEST.COM
Sep 25 22:36:27 dist-test-slave-wjdr krb5kdc[13210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375787, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.246.252@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM
I20260925 22:36:27.881794 13976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:27.868524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35362 (local address 127.6.246.252:46735)
0925 22:36:27.868947 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:27.868963 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:27.869002 (+ 39us) server_negotiation.cc:407] Connection header received
0925 22:36:27.869822 (+ 820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:27.869843 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:27.870217 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:27.870553 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:27.871200 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:27.872716 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:27.873576 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:27.874060 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:27.874229 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:27.877244 (+ 3015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:27.877249 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:27.877258 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:27.877282 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:27.879039 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:27.879514 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:27.879519 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:27.879525 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:27.879675 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:27.879941 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:27.879946 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:27.879951 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:27.880347 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:27.880563 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:27.880871 (+ 308us) server_negotiation.cc:299] Negotiation successful
0925 22:36:27.881144 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":124,"threads_started":1}
I20260925 22:36:27.884858 13294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:27.874958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53076 (local address 127.6.246.254:44671)
0925 22:36:27.875084 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:27.875088 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:27.875359 (+ 271us) server_negotiation.cc:407] Connection header received
0925 22:36:27.875521 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:27.875524 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:27.875565 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:27.875634 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:27.876410 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:27.877263 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:27.878555 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:27.878760 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:27.878845 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:27.881188 (+ 2343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:27.881191 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:27.881198 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:27.881220 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:27.883109 (+ 1889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:27.883651 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:27.883655 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:27.883657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:27.883735 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:27.884002 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:27.884007 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:27.884008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:27.884167 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:27.884268 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:27.884592 (+ 324us) server_negotiation.cc:299] Negotiation successful
0925 22:36:27.884694 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 22:36:27.884868 13977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:27.874772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:44671 (local address 127.0.0.1:53076)
0925 22:36:27.875215 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:27.875272 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:27.875421 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:27.875716 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:27.875723 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:27.876154 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:27.876276 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:27.876281 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:27.877375 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:27.877378 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:27.878437 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:27.878450 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:27.878869 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:27.878873 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:27.879006 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:27.879477 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:27.879495 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:27.881061 (+ 1566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:36:27.883237 (+ 2176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:27.883247 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:27.883251 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:27.883567 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:27.883825 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:27.883828 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:27.883831 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:27.883916 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:27.884266 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:36:27.884272 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:36:27.884381 (+ 109us) client_negotiation.cc:769] Sending connection context
0925 22:36:27.884530 (+ 149us) client_negotiation.cc:240] Negotiation successful
0925 22:36:27.884695 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":174,"threads_started":1}
I20260925 22:36:27.885893 13974 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.246.253" port: 40799 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 22:36:27 dist-test-slave-wjdr krb5kdc[13210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375787, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.246.252@KRBTEST.COM for oryx/127.6.246.253@KRBTEST.COM
I20260925 22:36:27.893707 13977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:27.886313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.253:40799 (local address 127.0.0.1:57966)
0925 22:36:27.886414 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:27.886427 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:27.886510 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:27.887282 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:27.887284 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:27.887431 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:27.887511 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:27.887516 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:27.888503 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:27.888506 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:27.889058 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:27.889066 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:27.889498 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:27.889500 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:27.889567 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:27.889966 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:27.889976 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:27.891058 (+ 1082us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:36:27.892602 (+ 1544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:27.892605 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:27.892606 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:27.892779 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:27.893018 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:27.893020 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:27.893021 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:27.893087 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:27.893380 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:36:27.893382 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:36:27.893446 (+ 64us) client_negotiation.cc:769] Sending connection context
0925 22:36:27.893519 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 22:36:27.893607 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 22:36:27.893770 13978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:27.886484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57966 (local address 127.6.246.253:40799)
0925 22:36:27.886711 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:27.886715 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:27.886726 (+ 11us) server_negotiation.cc:407] Connection header received
0925 22:36:27.886761 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:27.886763 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:27.886799 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:27.887259 (+ 460us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:27.887603 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:27.888373 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:27.889169 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:27.889399 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:27.889477 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:27.891187 (+ 1710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:27.891190 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:27.891192 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:27.891217 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:27.892511 (+ 1294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:27.892864 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:27.892866 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:27.892867 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:27.892950 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:27.893154 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:27.893156 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:27.893157 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:27.893308 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:27.893382 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:27.893551 (+ 169us) server_negotiation.cc:299] Negotiation successful
0925 22:36:27.893637 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":86,"threads_started":1}
I20260925 22:36:27.894539 13974 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.246.252" port: 46735 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 22:36:27 dist-test-slave-wjdr krb5kdc[13210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375787, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.246.252@KRBTEST.COM for oryx/127.6.246.252@KRBTEST.COM
I20260925 22:36:27.902238 13977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:27.894922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.252:46735 (local address 127.0.0.1:35368)
0925 22:36:27.895047 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:27.895061 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:27.895134 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:27.895348 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:27.895351 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:27.895493 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:27.895544 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:27.895548 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:27.896678 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:27.896681 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:27.897156 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:27.897162 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:27.897568 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:27.897571 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:27.897623 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:27.897946 (+ 323us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:27.897956 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:27.899183 (+ 1227us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:36:27.901119 (+ 1936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:27.901121 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:27.901123 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:27.901319 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:27.901548 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:27.901550 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:27.901551 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:27.901617 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:27.901905 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:36:27.901907 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:36:27.901972 (+ 65us) client_negotiation.cc:769] Sending connection context
0925 22:36:27.902045 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 22:36:27.902136 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260925 22:36:27.902318 13976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:27.895011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35368 (local address 127.6.246.252:46735)
0925 22:36:27.895112 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:27.895116 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:27.895128 (+ 12us) server_negotiation.cc:407] Connection header received
0925 22:36:27.895207 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:27.895211 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:27.895247 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:27.895327 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:27.895640 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:27.896551 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:27.897259 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:27.897482 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:27.897557 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:27.899318 (+ 1761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:27.899321 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:27.899324 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:27.899361 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:27.900947 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:27.901399 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:27.901402 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:27.901403 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:27.901474 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:27.901671 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:27.901673 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:27.901674 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:27.901833 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:27.901914 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:27.902078 (+ 164us) server_negotiation.cc:299] Negotiation successful
0925 22:36:27.902192 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":36}
I20260925 22:36:27.905237 13974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bf1e94e512547acba5b020a5eab47f5: Bootstrap starting.
I20260925 22:36:27.907400 13974 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3bf1e94e512547acba5b020a5eab47f5: Neither blocks nor log segments found. Creating new log.
I20260925 22:36:27.908085 13974 log.cc:824] T 00000000000000000000000000000000 P 3bf1e94e512547acba5b020a5eab47f5: Log is configured to *not* fsync() on all Append() calls
I20260925 22:36:27.909889 13974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bf1e94e512547acba5b020a5eab47f5: No bootstrap required, opened a new log
I20260925 22:36:27.912910 13974 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3bf1e94e512547acba5b020a5eab47f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f241ea7630914735a806c4b2935b467d" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 44671 } } peers { permanent_uuid: "d481871900894bbdbed9d74795df7ece" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 40799 } } peers { permanent_uuid: "3bf1e94e512547acba5b020a5eab47f5" member_type: VOTER last_known_addr { host: "127.6.246.252" port: 46735 } }
I20260925 22:36:27.913163 13974 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3bf1e94e512547acba5b020a5eab47f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:36:27.913221 13974 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3bf1e94e512547acba5b020a5eab47f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bf1e94e512547acba5b020a5eab47f5, State: Initialized, Role: FOLLOWER
I20260925 22:36:27.913686 13974 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3bf1e94e512547acba5b020a5eab47f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f241ea7630914735a806c4b2935b467d" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 44671 } } peers { permanent_uuid: "d481871900894bbdbed9d74795df7ece" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 40799 } } peers { permanent_uuid: "3bf1e94e512547acba5b020a5eab47f5" member_type: VOTER last_known_addr { host: "127.6.246.252" port: 46735 } }
I20260925 22:36:27.914151 13980 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bf1e94e512547acba5b020a5eab47f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f241ea7630914735a806c4b2935b467d" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 44671 } } peers { permanent_uuid: "d481871900894bbdbed9d74795df7ece" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 40799 } } peers { permanent_uuid: "3bf1e94e512547acba5b020a5eab47f5" member_type: VOTER last_known_addr { host: "127.6.246.252" port: 46735 } } }
I20260925 22:36:27.914283 13980 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bf1e94e512547acba5b020a5eab47f5 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:36:27.915032 13974 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3bf1e94e512547acba5b020a5eab47f5 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 22:36:27.919736 13991 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3bf1e94e512547acba5b020a5eab47f5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:36:27.919821 13991 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:36:27.923094 13828 master_runner.cc:428] Detected that this master 3bf1e94e512547acba5b020a5eab47f5 is joining an existing cluster
I20260925 22:36:27.923144 13828 master_runner.cc:432] Initiating AddMaster RPC to add 127.6.246.252:46735
I20260925 22:36:27.933918 13446 heartbeater.cc:503] Master 127.6.246.254:44671 was elected leader, sending a full tablet report...
I20260925 22:36:27.935250 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 13317
I20260925 22:36:27.941857 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.193:42341
--local_ip_for_outbound_sockets=127.6.246.193
--tserver_master_addrs=127.6.246.254:44671
--webserver_port=38061
--webserver_interface=127.6.246.193
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/krb5kdc/oryx_127.6.246.193.keytab
--principal=oryx/127.6.246.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:39015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.6.246.254:44671,127.6.246.253:40799
--tserver_master_addrs=127.6.246.254:44671,127.6.246.253:40799,127.6.246.252:46735 with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:36:27.951084 13976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:27.925650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35372 (local address 127.6.246.252:46735)
0925 22:36:27.925749 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:27.925753 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:27.928715 (+ 2962us) server_negotiation.cc:407] Connection header received
0925 22:36:27.928844 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:27.928846 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:27.928876 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:27.928914 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:27.929432 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:27.930079 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:27.931212 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:27.931412 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:27.931513 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:27.932725 (+ 1212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:27.932728 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:27.932731 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:27.932755 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:27.934972 (+ 2217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:27.949482 (+ 14510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:27.949486 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:27.949487 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:27.949557 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:27.950241 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:27.950244 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:27.950246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:27.950457 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:27.950533 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:27.950793 (+ 260us) server_negotiation.cc:299] Negotiation successful
0925 22:36:27.950912 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":31,"spinlock_wait_cycles":1920}
I20260925 22:36:27.951380 13978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:27.925956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57976 (local address 127.6.246.253:40799)
0925 22:36:27.926077 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:27.926080 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:27.926092 (+ 12us) server_negotiation.cc:407] Connection header received
0925 22:36:27.926465 (+ 373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:27.926468 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:27.926519 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:27.926602 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:27.927086 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:27.927787 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:27.928761 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:27.929148 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:27.929226 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:27.930723 (+ 1497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:27.930727 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:27.930730 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:27.930753 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:27.932892 (+ 2139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:27.949658 (+ 16766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:27.949664 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:27.949665 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:27.949794 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:27.950175 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:27.950181 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:27.950186 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:27.950461 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:27.950548 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:27.950989 (+ 441us) server_negotiation.cc:299] Negotiation successful
0925 22:36:27.951149 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 22:36:27.952041 13999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:27.925200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.253:40799 (local address 127.0.0.1:57976)
0925 22:36:27.925879 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:27.925892 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:27.926359 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:27.926632 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:27.926635 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:27.926846 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:27.926960 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:27.926969 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:27.927884 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:27.927888 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:27.928617 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:27.928627 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:27.929399 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:27.929405 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:27.929542 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:27.930134 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:27.930153 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:27.930580 (+ 427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:36:27.932990 (+ 2410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:27.932995 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:27.932997 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:27.949458 (+ 16461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:27.949941 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:27.949946 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:27.949950 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:27.950054 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:27.950635 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:36:27.950640 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:36:27.950759 (+ 119us) client_negotiation.cc:769] Sending connection context
0925 22:36:27.951586 (+ 827us) client_negotiation.cc:240] Negotiation successful
0925 22:36:27.951774 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"mutex_wait_us":73,"thread_start_us":100,"threads_started":1}
I20260925 22:36:27.953689 13998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:27.925616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.252:46735 (local address 127.0.0.1:35372)
0925 22:36:27.928645 (+ 3029us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:27.928660 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:27.928763 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:27.929012 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:27.929014 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:27.929213 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:27.929327 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:27.929336 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:27.930202 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:27.930207 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:27.931078 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:27.931085 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:27.931517 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:27.931520 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:27.931673 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:27.932105 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:27.932116 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:27.932599 (+ 483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:36:27.935077 (+ 2478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:27.935080 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:27.935082 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:27.949359 (+ 14277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:27.949650 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:27.949653 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:27.949655 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:27.949727 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:27.950573 (+ 846us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:36:27.950576 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:36:27.950644 (+ 68us) client_negotiation.cc:769] Sending connection context
0925 22:36:27.951643 (+ 999us) client_negotiation.cc:240] Negotiation successful
0925 22:36:27.953524 (+ 1881us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2947,"mutex_wait_us":1735,"thread_start_us":84,"threads_started":1}
I20260925 22:36:27.955430 13294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:27.925944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53078 (local address 127.6.246.254:44671)
0925 22:36:27.926066 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:27.926070 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:27.926083 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:36:27.926123 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:27.926126 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:27.926173 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:27.926252 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:27.926789 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:27.927551 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:27.928894 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:27.929231 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:27.929308 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:27.951819 (+ 22511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:27.951823 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:27.951826 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:27.951857 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:27.953585 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:27.954111 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:27.954115 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:27.954118 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:27.954211 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:27.954498 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:27.954501 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:27.954503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:27.954722 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:27.954839 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:27.955149 (+ 310us) server_negotiation.cc:299] Negotiation successful
0925 22:36:27.955273 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 22:36:27.955463 13997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:27.925163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:44671 (local address 127.0.0.1:53078)
0925 22:36:27.925438 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:27.925454 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:27.925556 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:27.926295 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:27.926297 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:27.926526 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:27.926670 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:27.926680 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:27.927667 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:27.927670 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:27.928750 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:27.928762 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:27.929474 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:27.929477 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:27.929596 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:27.949074 (+ 19478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:27.949093 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:27.950088 (+ 995us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:36:27.953735 (+ 3647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:27.953738 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:27.953740 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:27.953983 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:27.954307 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:27.954310 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:27.954312 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:27.954402 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:27.954880 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:36:27.954883 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:36:27.954967 (+ 84us) client_negotiation.cc:769] Sending connection context
0925 22:36:27.955079 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 22:36:27.955289 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":25,"thread_start_us":122,"threads_started":1}
Sep 25 22:36:27 dist-test-slave-wjdr krb5kdc[13210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375786, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.246.254@KRBTEST.COM for oryx/127.6.246.252@KRBTEST.COM
I20260925 22:36:27.967176 13976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:27.959274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35382 (local address 127.6.246.252:46735)
0925 22:36:27.959394 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:27.959398 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:27.959469 (+ 71us) server_negotiation.cc:407] Connection header received
0925 22:36:27.959654 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:27.959657 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:27.959707 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:27.959770 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:27.960244 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:27.960788 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:27.961983 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:27.962241 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:27.962346 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:27.964184 (+ 1838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:27.964188 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:27.964190 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:27.964212 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:27.965665 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:27.966047 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:27.966051 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:27.966053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:27.966123 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:27.966371 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:27.966376 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:27.966378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:27.966528 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:27.966622 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:27.966870 (+ 248us) server_negotiation.cc:299] Negotiation successful
0925 22:36:27.966958 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 22:36:27.967127 13812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:27.959235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.252:46735 (local address 127.0.0.1:35382)
0925 22:36:27.959417 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:27.959430 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:27.959524 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:27.959883 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:27.959886 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:27.960068 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:27.960144 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:27.960150 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:27.960946 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:27.960949 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:27.961863 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:27.961873 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:27.962348 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:27.962351 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:27.962443 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:27.962874 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:27.962887 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:27.964063 (+ 1176us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:36:27.965763 (+ 1700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:27.965768 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:27.965770 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:27.965958 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:27.966205 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:27.966209 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:27.966211 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:27.966291 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:27.966613 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:36:27.966616 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:36:27.966706 (+ 90us) client_negotiation.cc:769] Sending connection context
0925 22:36:27.966820 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 22:36:27.966959 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106}
I20260925 22:36:27.968493 13255 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.6.246.252:46735: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "3bf1e94e512547acba5b020a5eab47f5"
member_type: NON_VOTER
last_known_addr {
host: "127.6.246.252"
port: 46735
}
attrs {
promote: true
}
}
dest_uuid: "f241ea7630914735a806c4b2935b467d"
cas_config_opid_index: 5
I20260925 22:36:27.969121 13255 consensus_queue.cc:237] T 00000000000000000000000000000000 P f241ea7630914735a806c4b2935b467d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f241ea7630914735a806c4b2935b467d" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 44671 } } peers { permanent_uuid: "d481871900894bbdbed9d74795df7ece" member_type: NON_VOTER last_known_addr { host: "127.6.246.253" port: 40799 } attrs { promote: true } } peers { permanent_uuid: "3bf1e94e512547acba5b020a5eab47f5" member_type: NON_VOTER last_known_addr { host: "127.6.246.252" port: 46735 } attrs { promote: true } }
I20260925 22:36:27.970122 13815 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f241ea7630914735a806c4b2935b467d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 3bf1e94e512547acba5b020a5eab47f5 (127.6.246.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f241ea7630914735a806c4b2935b467d" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 44671 } } peers { permanent_uuid: "d481871900894bbdbed9d74795df7ece" member_type: NON_VOTER last_known_addr { host: "127.6.246.253" port: 40799 } attrs { promote: true } } peers { permanent_uuid: "3bf1e94e512547acba5b020a5eab47f5" member_type: NON_VOTER last_known_addr { host: "127.6.246.252" port: 46735 } attrs { promote: true } } }
I20260925 22:36:27.970745 13948 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3bf1e94e512547acba5b020a5eab47f5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:36:27.971527 13815 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.6.246.252:46735
I20260925 22:36:27.971699 13815 sys_catalog.cc:455] T 00000000000000000000000000000000 P f241ea7630914735a806c4b2935b467d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f241ea7630914735a806c4b2935b467d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f241ea7630914735a806c4b2935b467d" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 44671 } } peers { permanent_uuid: "d481871900894bbdbed9d74795df7ece" member_type: NON_VOTER last_known_addr { host: "127.6.246.253" port: 40799 } attrs { promote: true } } peers { permanent_uuid: "3bf1e94e512547acba5b020a5eab47f5" member_type: NON_VOTER last_known_addr { host: "127.6.246.252" port: 46735 } attrs { promote: true } } }
I20260925 22:36:27.971812 13815 sys_catalog.cc:458] T 00000000000000000000000000000000 P f241ea7630914735a806c4b2935b467d [sys.catalog]: This master's current role is: LEADER
I20260925 22:36:27.972193 13948 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3bf1e94e512547acba5b020a5eab47f5 [term 1 FOLLOWER]: Refusing update from remote peer f241ea7630914735a806c4b2935b467d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 22:36:27.972769 13828 master.cc:537] Master@127.6.246.252:46735 shutting down...
I20260925 22:36:27.972765 13816 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f241ea7630914735a806c4b2935b467d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bf1e94e512547acba5b020a5eab47f5" member_type: NON_VOTER last_known_addr { host: "127.6.246.252" port: 46735 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 22:36:27.974504 13239 consensus_peers.cc:487] T 00000000000000000000000000000000 P f241ea7630914735a806c4b2935b467d -> Peer 3bf1e94e512547acba5b020a5eab47f5 (127.6.246.252:46735): Couldn't send request to peer 3bf1e94e512547acba5b020a5eab47f5. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 22:36:27.975809 13978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:27.970482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57986 (local address 127.6.246.253:40799)
0925 22:36:27.970582 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:27.970586 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:27.970847 (+ 261us) server_negotiation.cc:407] Connection header received
0925 22:36:27.971112 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:27.971115 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:27.971194 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:27.971256 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:36:27.971629 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:27.972395 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:27.974355 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:27.974990 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:27.975069 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:27.975222 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:27.975638 (+ 416us) server_negotiation.cc:299] Negotiation successful
0925 22:36:27.975702 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 22:36:27.975809 13812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:27.970281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.253:40799 (local address 127.0.0.1:57986)
0925 22:36:27.970762 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:27.970791 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:27.970963 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:27.971401 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:27.971404 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:27.971418 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:36:27.971491 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:27.971496 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:27.973150 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:27.973153 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:27.974223 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:27.974235 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:27.975367 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:27.975370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:27.975495 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:27.975535 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 22:36:27.975623 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 22:36:27.975688 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406}
I20260925 22:36:27.976294 13866 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece [term 1 LEARNER]: Refusing update from remote peer f241ea7630914735a806c4b2935b467d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 22:36:27.976725 13816 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f241ea7630914735a806c4b2935b467d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d481871900894bbdbed9d74795df7ece" member_type: NON_VOTER last_known_addr { host: "127.6.246.253" port: 40799 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260925 22:36:27.977593 13828 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 3bf1e94e512547acba5b020a5eab47f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:36:27.977834 13828 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 3bf1e94e512547acba5b020a5eab47f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:36:27.977924 13828 tablet_replica.cc:330] T 00000000000000000000000000000000 P 3bf1e94e512547acba5b020a5eab47f5: stopping tablet replica
I20260925 22:36:27.978571 13866 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 3bf1e94e512547acba5b020a5eab47f5 (127.6.246.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f241ea7630914735a806c4b2935b467d" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 44671 } } peers { permanent_uuid: "d481871900894bbdbed9d74795df7ece" member_type: NON_VOTER last_known_addr { host: "127.6.246.253" port: 40799 } attrs { promote: true } } peers { permanent_uuid: "3bf1e94e512547acba5b020a5eab47f5" member_type: NON_VOTER last_known_addr { host: "127.6.246.252" port: 46735 } attrs { promote: true } } }
I20260925 22:36:27.980170 13895 sys_catalog.cc:455] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece [sys.catalog]: SysCatalogTable state changed. Reason: New leader f241ea7630914735a806c4b2935b467d. Latest consensus state: current_term: 1 leader_uuid: "f241ea7630914735a806c4b2935b467d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f241ea7630914735a806c4b2935b467d" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 44671 } } peers { permanent_uuid: "d481871900894bbdbed9d74795df7ece" member_type: NON_VOTER last_known_addr { host: "127.6.246.253" port: 40799 } attrs { promote: true } } peers { permanent_uuid: "3bf1e94e512547acba5b020a5eab47f5" member_type: NON_VOTER last_known_addr { host: "127.6.246.252" port: 46735 } attrs { promote: true } } }
I20260925 22:36:27.980278 13895 sys_catalog.cc:458] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece [sys.catalog]: This master's current role is: LEARNER
I20260925 22:36:27.980376 13895 sys_catalog.cc:455] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f241ea7630914735a806c4b2935b467d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f241ea7630914735a806c4b2935b467d" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 44671 } } peers { permanent_uuid: "d481871900894bbdbed9d74795df7ece" member_type: NON_VOTER last_known_addr { host: "127.6.246.253" port: 40799 } attrs { promote: true } } peers { permanent_uuid: "3bf1e94e512547acba5b020a5eab47f5" member_type: NON_VOTER last_known_addr { host: "127.6.246.252" port: 46735 } attrs { promote: true } } }
I20260925 22:36:27.980453 13895 sys_catalog.cc:458] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece [sys.catalog]: This master's current role is: LEARNER
I20260925 22:36:27.980652 13816 raft_consensus.cc:1050] T 00000000000000000000000000000000 P f241ea7630914735a806c4b2935b467d: attempting to promote NON_VOTER d481871900894bbdbed9d74795df7ece to VOTER
I20260925 22:36:27.980981 13816 consensus_queue.cc:237] T 00000000000000000000000000000000 P f241ea7630914735a806c4b2935b467d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f241ea7630914735a806c4b2935b467d" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 44671 } } peers { permanent_uuid: "d481871900894bbdbed9d74795df7ece" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 40799 } attrs { promote: false } } peers { permanent_uuid: "3bf1e94e512547acba5b020a5eab47f5" member_type: NON_VOTER last_known_addr { host: "127.6.246.252" port: 46735 } attrs { promote: true } }
I20260925 22:36:27.982020 13866 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece [term 1 LEARNER]: Refusing update from remote peer f241ea7630914735a806c4b2935b467d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260925 22:36:27.982355 13816 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f241ea7630914735a806c4b2935b467d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d481871900894bbdbed9d74795df7ece" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 40799 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260925 22:36:27.984328 13239 connection.cc:535] client connection to 127.6.246.252:46735 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260925 22:36:27.984423 13239 consensus_peers.cc:487] T 00000000000000000000000000000000 P f241ea7630914735a806c4b2935b467d -> Peer 3bf1e94e512547acba5b020a5eab47f5 (127.6.246.252:46735): Couldn't send request to peer 3bf1e94e512547acba5b020a5eab47f5. Status: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
I20260925 22:36:27.985486 13815 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f241ea7630914735a806c4b2935b467d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d481871900894bbdbed9d74795df7ece (127.6.246.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f241ea7630914735a806c4b2935b467d" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 44671 } } peers { permanent_uuid: "d481871900894bbdbed9d74795df7ece" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 40799 } attrs { promote: false } } peers { permanent_uuid: "3bf1e94e512547acba5b020a5eab47f5" member_type: NON_VOTER last_known_addr { host: "127.6.246.252" port: 46735 } attrs { promote: true } } }
I20260925 22:36:27.986708 13815 sys_catalog.cc:455] T 00000000000000000000000000000000 P f241ea7630914735a806c4b2935b467d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f241ea7630914735a806c4b2935b467d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f241ea7630914735a806c4b2935b467d" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 44671 } } peers { permanent_uuid: "d481871900894bbdbed9d74795df7ece" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 40799 } attrs { promote: false } } peers { permanent_uuid: "3bf1e94e512547acba5b020a5eab47f5" member_type: NON_VOTER last_known_addr { host: "127.6.246.252" port: 46735 } attrs { promote: true } } }
I20260925 22:36:27.986814 13815 sys_catalog.cc:458] T 00000000000000000000000000000000 P f241ea7630914735a806c4b2935b467d [sys.catalog]: This master's current role is: LEADER
I20260925 22:36:27.986948 13815 sys_catalog.cc:455] T 00000000000000000000000000000000 P f241ea7630914735a806c4b2935b467d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f241ea7630914735a806c4b2935b467d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f241ea7630914735a806c4b2935b467d" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 44671 } } peers { permanent_uuid: "d481871900894bbdbed9d74795df7ece" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 40799 } attrs { promote: false } } peers { permanent_uuid: "3bf1e94e512547acba5b020a5eab47f5" member_type: NON_VOTER last_known_addr { host: "127.6.246.252" port: 46735 } attrs { promote: true } } }
I20260925 22:36:27.987025 13815 sys_catalog.cc:458] T 00000000000000000000000000000000 P f241ea7630914735a806c4b2935b467d [sys.catalog]: This master's current role is: LEADER
I20260925 22:36:27.987594 13866 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d481871900894bbdbed9d74795df7ece (127.6.246.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f241ea7630914735a806c4b2935b467d" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 44671 } } peers { permanent_uuid: "d481871900894bbdbed9d74795df7ece" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 40799 } attrs { promote: false } } peers { permanent_uuid: "3bf1e94e512547acba5b020a5eab47f5" member_type: NON_VOTER last_known_addr { host: "127.6.246.252" port: 46735 } attrs { promote: true } } }
I20260925 22:36:27.988871 14002 sys_catalog.cc:455] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f241ea7630914735a806c4b2935b467d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f241ea7630914735a806c4b2935b467d" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 44671 } } peers { permanent_uuid: "d481871900894bbdbed9d74795df7ece" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 40799 } attrs { promote: false } } peers { permanent_uuid: "3bf1e94e512547acba5b020a5eab47f5" member_type: NON_VOTER last_known_addr { host: "127.6.246.252" port: 46735 } attrs { promote: true } } }
I20260925 22:36:27.988981 14002 sys_catalog.cc:458] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:36:27.996737 13828 master.cc:552] Master@127.6.246.252:46735 shutdown complete.
I20260925 22:36:27.996821 13828 master_runner.cc:305] Clearing existing system tablet
I20260925 22:36:27.997052 13828 fs_manager.cc:359] Metadata directory not provided
I20260925 22:36:27.997118 13828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:36:27.998554 13828 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 22:36:27.999486 14004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:27.999734 13828 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:36:27.999892 13828 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 22:36:27.999934 13828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/master-2/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/master-2/wal
uuid: "3bf1e94e512547acba5b020a5eab47f5"
format_stamp: "Formatted at 2026-09-25 22:36:27 on dist-test-slave-wjdr"
server_key: "3f255fd19ca85953ed35ba2f4f71ed25"
server_key_iv: "829da5a489e5e50393bda36e9c4c7787"
server_key_version: "encryptionkey@0"
I20260925 22:36:28.001713 13828 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 3bf1e94e512547acba5b020a5eab47f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:36:28.003710 13828 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 3bf1e94e512547acba5b020a5eab47f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 22:36:28.003819 13828 log.cc:1196] T 00000000000000000000000000000000 P 3bf1e94e512547acba5b020a5eab47f5: Deleting WAL directory at /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260925 22:36:28.004108 13828 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 3bf1e94e512547acba5b020a5eab47f5: Deleting consensus metadata
I20260925 22:36:28.004560 13828 master_runner.cc:315] Copying system tablet from 127.6.246.254:44671
I20260925 22:36:28.005653 13828 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 3bf1e94e512547acba5b020a5eab47f5: tablet copy: Beginning tablet copy session from remote peer at address 127.6.246.254:44671
I20260925 22:36:28.013361 13294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:28.006274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53088 (local address 127.6.246.254:44671)
0925 22:36:28.006417 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:28.006422 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:28.006611 (+ 189us) server_negotiation.cc:407] Connection header received
0925 22:36:28.006749 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:28.006752 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:28.006808 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:28.006892 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:28.007341 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:28.007879 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:28.008828 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:28.009058 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:28.009166 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:28.010277 (+ 1111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:28.010281 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:28.010284 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:28.010314 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:28.011728 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:28.012153 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:28.012158 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:28.012160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:28.012238 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:28.012555 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:28.012559 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:28.012560 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:28.012733 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:28.012838 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:28.013095 (+ 257us) server_negotiation.cc:299] Negotiation successful
0925 22:36:28.013204 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 22:36:28.013438 14010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:28.006200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:44671 (local address 127.0.0.1:53088)
0925 22:36:28.006512 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:28.006532 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:28.006658 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:28.006920 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:28.006924 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:28.007127 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:28.007223 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:28.007228 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:28.008026 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:28.008030 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:28.008696 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:28.008705 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:28.009169 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:28.009172 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:28.009282 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:28.009799 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:28.009811 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:28.010124 (+ 313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:36:28.011846 (+ 1722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:28.011849 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:28.011851 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:28.012048 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:28.012333 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:28.012338 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:28.012340 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:28.012424 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:28.012836 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:36:28.012839 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:36:28.012933 (+ 94us) client_negotiation.cc:769] Sending connection context
0925 22:36:28.013033 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 22:36:28.013168 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":101,"threads_started":1}
I20260925 22:36:28.014102 13275 tablet_copy_service.cc:140] P f241ea7630914735a806c4b2935b467d: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 3bf1e94e512547acba5b020a5eab47f5 ({username='oryx', principal='oryx/127.6.246.252@KRBTEST.COM'} at 127.0.0.1:53088)
I20260925 22:36:28.014206 13275 tablet_copy_service.cc:161] P f241ea7630914735a806c4b2935b467d: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 3bf1e94e512547acba5b020a5eab47f5 at {username='oryx', principal='oryx/127.6.246.252@KRBTEST.COM'} at 127.0.0.1:53088: session id = 3bf1e94e512547acba5b020a5eab47f5-00000000000000000000000000000000
I20260925 22:36:28.015229 13275 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P f241ea7630914735a806c4b2935b467d: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 22:36:28.016577 13828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 22:36:28.018527 13828 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 3bf1e94e512547acba5b020a5eab47f5: tablet copy: Starting download of 0 data blocks...
I20260925 22:36:28.018710 13828 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 3bf1e94e512547acba5b020a5eab47f5: tablet copy: Starting download of 1 WAL segments...
I20260925 22:36:28.019984 13828 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 3bf1e94e512547acba5b020a5eab47f5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 22:36:28.021435 13275 tablet_copy_service.cc:342] P f241ea7630914735a806c4b2935b467d: Request end of tablet copy session 3bf1e94e512547acba5b020a5eab47f5-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.6.246.252@KRBTEST.COM'} at 127.0.0.1:53088
I20260925 22:36:28.021522 13275 tablet_copy_service.cc:433] P f241ea7630914735a806c4b2935b467d: ending tablet copy session 3bf1e94e512547acba5b020a5eab47f5-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 3bf1e94e512547acba5b020a5eab47f5
I20260925 22:36:28.027624 13828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:36:28.029289 14011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:28.029343 14012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:28.029624 14014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:28.032668 13828 server_base.cc:1034] running on GCE node
I20260925 22:36:28.032810 13828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:36:28.033039 13828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:36:28.034178 13828 hybrid_clock.cc:648] HybridClock initialized: now 1790375788034167 us; error 29 us; skew 500 ppm
Sep 25 22:36:28 dist-test-slave-wjdr krb5kdc[13210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375788, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.246.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:36:28.035588 13828 init.cc:377] Logged in from keytab as oryx/127.6.246.252@KRBTEST.COM (short username oryx)
I20260925 22:36:28.036069 13828 webserver.cc:460] Webserver started at http://127.6.246.252:46465/ using document root <none> and password file <none>
I20260925 22:36:28.036244 13828 fs_manager.cc:359] Metadata directory not provided
I20260925 22:36:28.036299 13828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:36:28.037380 13828 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 22:36:28.038213 14021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:28.038515 13828 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 22:36:28.038600 13828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/master-2/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/master-2/wal
uuid: "3bf1e94e512547acba5b020a5eab47f5"
format_stamp: "Formatted at 2026-09-25 22:36:27 on dist-test-slave-wjdr"
server_key: "3f255fd19ca85953ed35ba2f4f71ed25"
server_key_iv: "829da5a489e5e50393bda36e9c4c7787"
server_key_version: "encryptionkey@0"
I20260925 22:36:28.038681 13828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:36:28.048918 13828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:36:28.051031 13828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:36:28.055290 13828 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.252:46735
I20260925 22:36:28.055817 13828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/master-2/data/info.pb
I20260925 22:36:28.065577 14074 sys_catalog.cc:263] Verifying existing consensus state
I20260925 22:36:28.066866 14074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bf1e94e512547acba5b020a5eab47f5: Bootstrap starting.
I20260925 22:36:28.067435 14073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.252:46735 every 8 connection(s)
I20260925 22:36:28.076293 14074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bf1e94e512547acba5b020a5eab47f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 22:36:28.076614 14074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bf1e94e512547acba5b020a5eab47f5: Bootstrap complete.
I20260925 22:36:28.077224 14074 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3bf1e94e512547acba5b020a5eab47f5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f241ea7630914735a806c4b2935b467d" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 44671 } } peers { permanent_uuid: "d481871900894bbdbed9d74795df7ece" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 40799 } attrs { promote: false } } peers { permanent_uuid: "3bf1e94e512547acba5b020a5eab47f5" member_type: NON_VOTER last_known_addr { host: "127.6.246.252" port: 46735 } attrs { promote: true } }
I20260925 22:36:28.077355 14074 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3bf1e94e512547acba5b020a5eab47f5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3bf1e94e512547acba5b020a5eab47f5, State: Initialized, Role: LEARNER
I20260925 22:36:28.077471 14074 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3bf1e94e512547acba5b020a5eab47f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f241ea7630914735a806c4b2935b467d" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 44671 } } peers { permanent_uuid: "d481871900894bbdbed9d74795df7ece" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 40799 } attrs { promote: false } } peers { permanent_uuid: "3bf1e94e512547acba5b020a5eab47f5" member_type: NON_VOTER last_known_addr { host: "127.6.246.252" port: 46735 } attrs { promote: true } }
I20260925 22:36:28.077801 14077 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bf1e94e512547acba5b020a5eab47f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f241ea7630914735a806c4b2935b467d" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 44671 } } peers { permanent_uuid: "d481871900894bbdbed9d74795df7ece" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 40799 } attrs { promote: false } } peers { permanent_uuid: "3bf1e94e512547acba5b020a5eab47f5" member_type: NON_VOTER last_known_addr { host: "127.6.246.252" port: 46735 } attrs { promote: true } } }
I20260925 22:36:28.077972 14074 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3bf1e94e512547acba5b020a5eab47f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:36:28.077997 14077 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bf1e94e512547acba5b020a5eab47f5 [sys.catalog]: This master's current role is: LEARNER
I20260925 22:36:28.081466 14088 catalog_manager.cc:1259] Loaded cluster ID: 2f47f719071c41b3ad8263c885d14031
I20260925 22:36:28.081530 14088 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3bf1e94e512547acba5b020a5eab47f5: loading cluster ID for follower catalog manager: success
I20260925 22:36:28.083001 14088 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3bf1e94e512547acba5b020a5eab47f5: acquiring CA information for follower catalog manager: success
I20260925 22:36:28.084069 14088 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3bf1e94e512547acba5b020a5eab47f5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 22:36:28.101074 14000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:36:28.101401 14000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:36:28.101462 14000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:36:28.107553 14000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:36:28.107628 14000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:36:28.107712 14000 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.193
I20260925 22:36:28.116065 14000 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:39015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/krb5kdc/oryx_127.6.246.193.keytab
--principal=oryx/127.6.246.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.193:42341
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.246.193
--webserver_port=38061
--tserver_master_addrs=127.6.246.254:44671,127.6.246.253:40799,127.6.246.252:46735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14000
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.193
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:36:28.117283 14000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:36:28.118429 14000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:36:28.129292 14094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:28.129302 14093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:28.129367 14000 server_base.cc:1034] running on GCE node
W20260925 22:36:28.129336 14096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:28.129912 14000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:36:28.130545 14000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:36:28.131721 14000 hybrid_clock.cc:648] HybridClock initialized: now 1790375788131703 us; error 27 us; skew 500 ppm
Sep 25 22:36:28 dist-test-slave-wjdr krb5kdc[13210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375788, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.246.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:36:28.134388 14000 init.cc:377] Logged in from keytab as oryx/127.6.246.193@KRBTEST.COM (short username oryx)
I20260925 22:36:28.135128 14000 webserver.cc:460] Webserver started at http://127.6.246.193:38061/ using document root <none> and password file <none>
I20260925 22:36:28.135617 14000 fs_manager.cc:359] Metadata directory not provided
I20260925 22:36:28.135684 14000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:36:28.139355 14000 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:36:28.141273 14103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:28.142441 14000 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:36:28.142546 14000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-0/wal
uuid: "67b78fd0bb194aa0b4b9cefa890e343f"
format_stamp: "Formatted at 2026-09-25 22:36:26 on dist-test-slave-wjdr"
server_key: "edd0397cf1c9470b14bce394471c93d2"
server_key_iv: "a5d16793b5211f9a9b18961053e8473a"
server_key_version: "encryptionkey@0"
I20260925 22:36:28.143025 14000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:36:28.146565 13580 heartbeater.cc:503] Master 127.6.246.254:44671 was elected leader, sending a full tablet report...
I20260925 22:36:28.174686 14000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:36:28.177911 14000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:36:28.178109 14000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:36:28.178699 14000 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:36:28.179531 14000 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:36:28.179597 14000 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:28.179669 14000 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:36:28.179716 14000 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:28.188858 14000 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.193:42341
I20260925 22:36:28.188897 14216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.193:42341 every 8 connection(s)
I20260925 22:36:28.189815 14000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
I20260925 22:36:28.192770 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 14000
I20260925 22:36:28.192979 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 13451
I20260925 22:36:28.200870 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.194:36135
--local_ip_for_outbound_sockets=127.6.246.194
--tserver_master_addrs=127.6.246.254:44671
--webserver_port=45597
--webserver_interface=127.6.246.194
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/krb5kdc/oryx_127.6.246.194.keytab
--principal=oryx/127.6.246.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:39015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.6.246.254:44671,127.6.246.253:40799
--tserver_master_addrs=127.6.246.254:44671,127.6.246.253:40799,127.6.246.252:46735 with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 22:36:28 dist-test-slave-wjdr krb5kdc[13210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375788, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.246.193@KRBTEST.COM for oryx/127.6.246.253@KRBTEST.COM
Sep 25 22:36:28 dist-test-slave-wjdr krb5kdc[13210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375788, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.246.193@KRBTEST.COM for oryx/127.6.246.252@KRBTEST.COM
Sep 25 22:36:28 dist-test-slave-wjdr krb5kdc[13210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375788, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.246.193@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM
I20260925 22:36:28.208128 13978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:28.191663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.193:52403 (local address 127.6.246.253:40799)
0925 22:36:28.191792 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:28.191796 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:28.195220 (+ 3424us) server_negotiation.cc:407] Connection header received
0925 22:36:28.195412 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:28.195416 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:28.195481 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:28.195541 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:28.197269 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:28.198102 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:28.200130 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:28.200460 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:28.200559 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:28.203919 (+ 3360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:28.203923 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:28.203925 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:28.203948 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:28.205655 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:28.206292 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:28.206299 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:28.206303 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:28.206390 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:28.206797 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:28.206802 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:28.206806 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:28.207030 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:28.207119 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:28.207842 (+ 723us) server_negotiation.cc:299] Negotiation successful
0925 22:36:28.207963 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 22:36:28.208194 13294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:28.191671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.193:38267 (local address 127.6.246.254:44671)
0925 22:36:28.191881 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:28.191886 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:28.193326 (+ 1440us) server_negotiation.cc:407] Connection header received
0925 22:36:28.194967 (+ 1641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:28.194974 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:28.195041 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:28.195132 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:28.197133 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:28.198095 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:28.200121 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:28.200520 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:28.200593 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:28.204860 (+ 4267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:28.204866 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:28.204872 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:28.204904 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:28.206563 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:28.207005 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:28.207008 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:28.207010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:28.207091 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:28.207386 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:28.207389 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:28.207390 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:28.207578 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:28.207656 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:28.207961 (+ 305us) server_negotiation.cc:299] Negotiation successful
0925 22:36:28.208062 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260925 22:36:28.209126 14224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:28.194024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.193:38943 (local address 127.6.246.252:46735)
0925 22:36:28.194265 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:28.194271 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:28.195625 (+ 1354us) server_negotiation.cc:407] Connection header received
0925 22:36:28.195678 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:28.195682 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:28.195733 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:28.195819 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:28.197313 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:28.198367 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:28.200103 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:28.200492 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:28.200575 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:28.204210 (+ 3635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:28.204215 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:28.204219 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:28.204245 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:28.206201 (+ 1956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:28.206724 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:28.206730 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:28.206735 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:28.206816 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:28.207109 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:28.207115 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:28.207119 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:28.207341 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:28.207445 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:28.208842 (+ 1397us) server_negotiation.cc:299] Negotiation successful
0925 22:36:28.208984 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":90,"threads_started":1}
I20260925 22:36:28.209251 14221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:28.192098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:44671 (local address 127.6.246.193:38267)
0925 22:36:28.193173 (+ 1075us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:28.193227 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:28.194720 (+ 1493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:28.195448 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:28.195460 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:28.196042 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:28.196945 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:28.196966 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:28.198260 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:28.198266 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:28.199989 (+ 1723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:28.200003 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:28.200753 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:28.200757 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:28.201010 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:28.201720 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:28.201743 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:28.204694 (+ 2951us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:36:28.206678 (+ 1984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:28.206681 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:28.206682 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:28.206901 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:28.207186 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:28.207189 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:28.207190 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:28.207266 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:28.207706 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:36:28.207708 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:36:28.207783 (+ 75us) client_negotiation.cc:769] Sending connection context
0925 22:36:28.208326 (+ 543us) client_negotiation.cc:240] Negotiation successful
0925 22:36:28.208436 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":703,"mutex_wait_us":421,"spinlock_wait_cycles":3328,"thread_start_us":555,"threads_started":1}
I20260925 22:36:28.209527 14222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:28.192101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.253:40799 (local address 127.6.246.193:52403)
0925 22:36:28.195136 (+ 3035us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:28.195151 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:28.195290 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:28.195694 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:28.195696 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:28.196070 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:28.196951 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:28.196967 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:28.198255 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:28.198258 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:28.199970 (+ 1712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:28.199977 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:28.200715 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:28.200718 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:28.201309 (+ 591us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:28.201929 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:28.201943 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:28.203691 (+ 1748us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:36:28.205761 (+ 2070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:28.205768 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:28.205792 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:28.206167 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:28.206500 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:28.206503 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:28.206505 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:28.206682 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:28.207172 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:36:28.207179 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:36:28.207457 (+ 278us) client_negotiation.cc:769] Sending connection context
0925 22:36:28.207643 (+ 186us) client_negotiation.cc:240] Negotiation successful
0925 22:36:28.207995 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2938,"mutex_wait_us":570,"spinlock_wait_cycles":26752,"thread_start_us":563,"threads_started":1}
I20260925 22:36:28.209856 14223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:28.193536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.252:46735 (local address 127.6.246.193:38943)
0925 22:36:28.195453 (+ 1917us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:28.195467 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:28.195546 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:28.195943 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:28.195949 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:28.196148 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:28.196948 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:28.196967 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:28.198489 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:28.198492 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:28.199967 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:28.199977 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:28.200715 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:28.200718 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:28.201011 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:28.201757 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:28.201767 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:28.204074 (+ 2307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:36:28.206325 (+ 2251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:28.206331 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:28.206336 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:28.206601 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:28.206914 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:28.206920 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:28.206925 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:28.207013 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:28.207491 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:36:28.207497 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:36:28.207602 (+ 105us) client_negotiation.cc:769] Sending connection context
0925 22:36:28.207710 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 22:36:28.209669 (+ 1959us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1846,"mutex_wait_us":407,"spinlock_wait_cycles":29696,"thread_start_us":83,"threads_started":1}
I20260925 22:36:28.210852 14217 heartbeater.cc:347] Connected to a master server at 127.6.246.252:46735
I20260925 22:36:28.211061 14217 heartbeater.cc:464] Registering TS with master...
I20260925 22:36:28.211273 14218 heartbeater.cc:347] Connected to a master server at 127.6.246.254:44671
I20260925 22:36:28.211347 14218 heartbeater.cc:464] Registering TS with master...
I20260925 22:36:28.211628 14217 heartbeater.cc:511] Master 127.6.246.252:46735 requested a full tablet report, sending...
I20260925 22:36:28.211628 14218 heartbeater.cc:511] Master 127.6.246.254:44671 requested a full tablet report, sending...
I20260925 22:36:28.211844 14219 heartbeater.cc:347] Connected to a master server at 127.6.246.253:40799
I20260925 22:36:28.211932 14219 heartbeater.cc:464] Registering TS with master...
I20260925 22:36:28.212116 14219 heartbeater.cc:511] Master 127.6.246.253:40799 requested a full tablet report, sending...
I20260925 22:36:28.212934 13255 ts_manager.cc:194] Re-registered known tserver with Master: 67b78fd0bb194aa0b4b9cefa890e343f (127.6.246.193:42341)
I20260925 22:36:28.213171 14034 ts_manager.cc:194] Registered new tserver with Master: 67b78fd0bb194aa0b4b9cefa890e343f (127.6.246.193:42341)
I20260925 22:36:28.213826 13255 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.246.193@KRBTEST.COM'} at 127.6.246.193:38267
I20260925 22:36:28.214897 13856 ts_manager.cc:194] Registered new tserver with Master: 67b78fd0bb194aa0b4b9cefa890e343f (127.6.246.193:42341)
W20260925 22:36:28.314734 14225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:36:28.314965 14225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:36:28.315007 14225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:36:28.319129 14225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:36:28.319177 14225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:36:28.319234 14225 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.194
I20260925 22:36:28.324184 14225 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:39015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/krb5kdc/oryx_127.6.246.194.keytab
--principal=oryx/127.6.246.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.194:36135
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.246.194
--webserver_port=45597
--tserver_master_addrs=127.6.246.254:44671,127.6.246.253:40799,127.6.246.252:46735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14225
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.194
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:36:28.325145 14225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:36:28.326077 14225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 22:36:28.335732 14225 server_base.cc:1034] running on GCE node
W20260925 22:36:28.335690 14231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:28.335651 14230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:28.335664 14233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:28.336274 14225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:36:28.336921 14225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:36:28.338078 14225 hybrid_clock.cc:648] HybridClock initialized: now 1790375788338061 us; error 28 us; skew 500 ppm
Sep 25 22:36:28 dist-test-slave-wjdr krb5kdc[13210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375788, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.246.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:36:28.340737 14225 init.cc:377] Logged in from keytab as oryx/127.6.246.194@KRBTEST.COM (short username oryx)
I20260925 22:36:28.341534 14225 webserver.cc:460] Webserver started at http://127.6.246.194:45597/ using document root <none> and password file <none>
I20260925 22:36:28.342068 14225 fs_manager.cc:359] Metadata directory not provided
I20260925 22:36:28.342162 14225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:36:28.345883 14225 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:36:28.346951 13714 heartbeater.cc:503] Master 127.6.246.254:44671 was elected leader, sending a full tablet report...
I20260925 22:36:28.347913 14240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:28.349188 14225 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:36:28.349285 14225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-1/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-1/wal
uuid: "8d3bf93a4c6844cf9013e229796c55d1"
format_stamp: "Formatted at 2026-09-25 22:36:27 on dist-test-slave-wjdr"
server_key: "d228e1a0fcfd9ca877cc88d3c0cf0b96"
server_key_iv: "df321dcab389123d026b57aa93b6a364"
server_key_version: "encryptionkey@0"
I20260925 22:36:28.349861 14225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:36:28.374909 14225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:36:28.377683 14225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:36:28.377863 14225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:36:28.378469 14225 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:36:28.379410 14225 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:36:28.379455 14225 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:28.379513 14225 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:36:28.379542 14225 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:28.388648 14225 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.194:36135
I20260925 22:36:28.388675 14353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.194:36135 every 8 connection(s)
I20260925 22:36:28.389601 14225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
I20260925 22:36:28.390223 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 14225
I20260925 22:36:28.390401 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 13585
I20260925 22:36:28.397943 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.195:44879
--local_ip_for_outbound_sockets=127.6.246.195
--tserver_master_addrs=127.6.246.254:44671
--webserver_port=38517
--webserver_interface=127.6.246.195
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/krb5kdc/oryx_127.6.246.195.keytab
--principal=oryx/127.6.246.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:39015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.6.246.254:44671,127.6.246.253:40799
--tserver_master_addrs=127.6.246.254:44671,127.6.246.253:40799,127.6.246.252:46735 with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 22:36:28 dist-test-slave-wjdr krb5kdc[13210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375788, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.246.194@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM
Sep 25 22:36:28 dist-test-slave-wjdr krb5kdc[13210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375788, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.246.194@KRBTEST.COM for oryx/127.6.246.252@KRBTEST.COM
Sep 25 22:36:28 dist-test-slave-wjdr krb5kdc[13210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375788, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.246.194@KRBTEST.COM for oryx/127.6.246.253@KRBTEST.COM
I20260925 22:36:28.409678 13978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:28.392937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.194:55305 (local address 127.6.246.253:40799)
0925 22:36:28.393231 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:28.393238 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:28.393489 (+ 251us) server_negotiation.cc:407] Connection header received
0925 22:36:28.394879 (+ 1390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:28.394886 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:28.394948 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:28.395007 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:28.397018 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:28.398162 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:28.399905 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:28.400282 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:28.400367 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:28.405074 (+ 4707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:28.405077 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:28.405080 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:28.405104 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:28.407066 (+ 1962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:28.407875 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:28.407878 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:28.407880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:28.407955 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:28.408366 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:28.408369 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:28.408371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:28.408620 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:28.408695 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:28.409401 (+ 706us) server_negotiation.cc:299] Negotiation successful
0925 22:36:28.409517 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190}
I20260925 22:36:28.409678 14224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:28.393061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.194:45051 (local address 127.6.246.252:46735)
0925 22:36:28.393209 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:28.393215 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:28.394076 (+ 861us) server_negotiation.cc:407] Connection header received
0925 22:36:28.394864 (+ 788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:28.394867 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:28.394921 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:28.394976 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:28.397015 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:28.397823 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:28.399875 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:28.400211 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:28.400287 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:28.405044 (+ 4757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:28.405048 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:28.405050 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:28.405075 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:28.407065 (+ 1990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:28.407841 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:28.407848 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:28.407853 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:28.407941 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:28.408373 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:28.408377 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:28.408378 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:28.408611 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:28.408686 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:28.409401 (+ 715us) server_negotiation.cc:299] Negotiation successful
0925 22:36:28.409517 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 22:36:28.409795 13294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:28.392049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.194:52975 (local address 127.6.246.254:44671)
0925 22:36:28.392184 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:28.392189 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:28.393844 (+ 1655us) server_negotiation.cc:407] Connection header received
0925 22:36:28.394869 (+ 1025us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:28.394872 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:28.394921 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:28.394976 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:28.397015 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:28.397823 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:28.399922 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:28.400230 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:28.400293 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:28.404784 (+ 4491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:28.404789 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:28.404795 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:28.404831 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:28.406967 (+ 2136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:28.407901 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:28.407907 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:28.407912 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:28.408009 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:28.408414 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:28.408417 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:28.408419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:28.408710 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:28.408810 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:28.409440 (+ 630us) server_negotiation.cc:299] Negotiation successful
0925 22:36:28.409575 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 22:36:28.411334 14358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:28.391867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.253:40799 (local address 127.6.246.194:55305)
0925 22:36:28.393318 (+ 1451us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:28.393375 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:28.394632 (+ 1257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:28.395372 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:28.395381 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:28.396050 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:28.396859 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:28.396873 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:28.398322 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:28.398328 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:28.399756 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:28.399769 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:28.400592 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:28.400598 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:28.400776 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:28.401796 (+ 1020us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:28.401816 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:28.404944 (+ 3128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:36:28.407185 (+ 2241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:28.407189 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:28.407191 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:28.407743 (+ 552us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:28.408057 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:28.408060 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:28.408062 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:28.408252 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:28.408764 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:36:28.408770 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:36:28.409118 (+ 348us) client_negotiation.cc:769] Sending connection context
0925 22:36:28.409911 (+ 793us) client_negotiation.cc:240] Negotiation successful
0925 22:36:28.410266 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":814,"mutex_wait_us":235,"spinlock_wait_cycles":24064,"thread_start_us":130,"threads_started":1}
I20260925 22:36:28.411348 14359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:28.391867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:44671 (local address 127.6.246.194:52975)
0925 22:36:28.393741 (+ 1874us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:28.393771 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:28.394636 (+ 865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:28.395370 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:28.395379 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:28.396070 (+ 691us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:28.396861 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:28.396869 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:28.398004 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:28.398008 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:28.399741 (+ 1733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:28.399750 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:28.400501 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:28.400505 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:28.400817 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:28.401639 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:28.401654 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:28.404570 (+ 2916us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:36:28.407111 (+ 2541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:28.407118 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:28.407136 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:28.407752 (+ 616us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:28.408135 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:28.408141 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:28.408146 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:28.408284 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:28.408879 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:36:28.408884 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:36:28.409123 (+ 239us) client_negotiation.cc:769] Sending connection context
0925 22:36:28.410067 (+ 944us) client_negotiation.cc:240] Negotiation successful
0925 22:36:28.410369 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1790,"mutex_wait_us":1035,"spinlock_wait_cycles":24192,"thread_start_us":100,"threads_started":1}
I20260925 22:36:28.411350 14360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:28.392938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.252:46735 (local address 127.6.246.194:45051)
0925 22:36:28.393999 (+ 1061us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:28.394008 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:28.394636 (+ 628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:28.395370 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:28.395379 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:28.396070 (+ 691us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:28.396846 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:28.396869 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:28.398004 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:28.398008 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:28.399748 (+ 1740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:28.399753 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:28.400501 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:28.400505 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:28.400813 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:28.401559 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:28.401574 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:28.404889 (+ 3315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:36:28.407186 (+ 2297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:28.407189 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:28.407191 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:28.407707 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:28.408057 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:28.408060 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:28.408062 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:28.408252 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:28.408764 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:36:28.408770 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:36:28.409118 (+ 348us) client_negotiation.cc:769] Sending connection context
0925 22:36:28.410327 (+ 1209us) client_negotiation.cc:240] Negotiation successful
0925 22:36:28.410478 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1016,"mutex_wait_us":499,"spinlock_wait_cycles":30592,"thread_start_us":77,"threads_started":1}
I20260925 22:36:28.412679 14356 heartbeater.cc:347] Connected to a master server at 127.6.246.253:40799
I20260925 22:36:28.412746 14354 heartbeater.cc:347] Connected to a master server at 127.6.246.252:46735
I20260925 22:36:28.412773 14355 heartbeater.cc:347] Connected to a master server at 127.6.246.254:44671
I20260925 22:36:28.412906 14354 heartbeater.cc:464] Registering TS with master...
I20260925 22:36:28.412906 14356 heartbeater.cc:464] Registering TS with master...
I20260925 22:36:28.412906 14355 heartbeater.cc:464] Registering TS with master...
I20260925 22:36:28.413506 14356 heartbeater.cc:511] Master 127.6.246.253:40799 requested a full tablet report, sending...
I20260925 22:36:28.413516 14354 heartbeater.cc:511] Master 127.6.246.252:46735 requested a full tablet report, sending...
I20260925 22:36:28.413516 14355 heartbeater.cc:511] Master 127.6.246.254:44671 requested a full tablet report, sending...
I20260925 22:36:28.414608 13255 ts_manager.cc:194] Re-registered known tserver with Master: 8d3bf93a4c6844cf9013e229796c55d1 (127.6.246.194:36135)
I20260925 22:36:28.414645 13856 ts_manager.cc:194] Registered new tserver with Master: 8d3bf93a4c6844cf9013e229796c55d1 (127.6.246.194:36135)
I20260925 22:36:28.414645 14034 ts_manager.cc:194] Registered new tserver with Master: 8d3bf93a4c6844cf9013e229796c55d1 (127.6.246.194:36135)
I20260925 22:36:28.415337 13255 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.246.194@KRBTEST.COM'} at 127.6.246.194:52975
W20260925 22:36:28.511452 14361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:36:28.511701 14361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:36:28.511740 14361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:36:28.515681 14361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:36:28.515730 14361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:36:28.515785 14361 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.195
I20260925 22:36:28.520670 14361 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:39015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/krb5kdc/oryx_127.6.246.195.keytab
--principal=oryx/127.6.246.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.195:44879
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.246.195
--webserver_port=38517
--tserver_master_addrs=127.6.246.254:44671,127.6.246.253:40799,127.6.246.252:46735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14361
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.195
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:36:28.521536 14361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:36:28.522424 14361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:36:28.531325 14369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:28.531347 14366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:28.531353 14361 server_base.cc:1034] running on GCE node
W20260925 22:36:28.531365 14367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:28.531931 14361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:36:28.532596 14361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:36:28.533779 14361 hybrid_clock.cc:648] HybridClock initialized: now 1790375788533769 us; error 32 us; skew 500 ppm
Sep 25 22:36:28 dist-test-slave-wjdr krb5kdc[13210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375788, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.246.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:36:28.536358 14361 init.cc:377] Logged in from keytab as oryx/127.6.246.195@KRBTEST.COM (short username oryx)
I20260925 22:36:28.537142 14361 webserver.cc:460] Webserver started at http://127.6.246.195:38517/ using document root <none> and password file <none>
I20260925 22:36:28.537654 14361 fs_manager.cc:359] Metadata directory not provided
I20260925 22:36:28.537751 14361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:36:28.541575 14361 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260925 22:36:28.543414 14376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:28.544564 14361 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:36:28.544677 14361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-2/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-2/wal
uuid: "d941f16284f0429381d37c34d0eaae2a"
format_stamp: "Formatted at 2026-09-25 22:36:27 on dist-test-slave-wjdr"
server_key: "4c850abc83584ab2d418603cac494190"
server_key_iv: "6b75aa2e350985d879c0209a33759e4c"
server_key_version: "encryptionkey@0"
I20260925 22:36:28.545173 14361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:36:28.560932 14361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:36:28.562171 14224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:28.558128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35396 (local address 127.6.246.252:46735)
0925 22:36:28.558275 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:28.558282 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:28.558318 (+ 36us) server_negotiation.cc:407] Connection header received
0925 22:36:28.558560 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:28.558562 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:28.558615 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:28.558694 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:36:28.559000 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:28.559582 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:28.561056 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:28.561615 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:28.561732 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:28.561863 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:28.561998 (+ 135us) server_negotiation.cc:299] Negotiation successful
0925 22:36:28.562057 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 22:36:28.562223 14382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:28.558032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.252:46735 (local address 127.0.0.1:35396)
0925 22:36:28.558250 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:28.558262 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:28.558385 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:28.558738 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:28.558742 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:28.558765 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:36:28.558866 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:28.558872 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:28.559678 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:28.559681 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:28.560906 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:28.560918 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:28.561757 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:28.561761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:28.561857 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:28.561894 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 22:36:28.561970 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 22:36:28.562054 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":63,"threads_started":1}
I20260925 22:36:28.562812 14047 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 3bf1e94e512547acba5b020a5eab47f5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260925 22:36:28.563181 14077 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bf1e94e512547acba5b020a5eab47f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f241ea7630914735a806c4b2935b467d. Latest consensus state: current_term: 1 leader_uuid: "f241ea7630914735a806c4b2935b467d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f241ea7630914735a806c4b2935b467d" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 44671 } } peers { permanent_uuid: "d481871900894bbdbed9d74795df7ece" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 40799 } attrs { promote: false } } peers { permanent_uuid: "3bf1e94e512547acba5b020a5eab47f5" member_type: NON_VOTER last_known_addr { host: "127.6.246.252" port: 46735 } attrs { promote: true } } }
I20260925 22:36:28.563320 14077 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bf1e94e512547acba5b020a5eab47f5 [sys.catalog]: This master's current role is: LEARNER
I20260925 22:36:28.564323 14361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:36:28.564517 14361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:36:28.565021 14361 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:36:28.565850 14361 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:36:28.565903 14361 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:28.565944 14361 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:36:28.565961 14361 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:28.575009 14361 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.195:44879
I20260925 22:36:28.575042 14491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.195:44879 every 8 connection(s)
I20260925 22:36:28.575917 14361 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
I20260925 22:36:28.584896 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 14361
Sep 25 22:36:28 dist-test-slave-wjdr krb5kdc[13210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375788, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.246.195@KRBTEST.COM for oryx/127.6.246.253@KRBTEST.COM
Sep 25 22:36:28 dist-test-slave-wjdr krb5kdc[13210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375788, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.246.195@KRBTEST.COM for oryx/127.6.246.252@KRBTEST.COM
Sep 25 22:36:28 dist-test-slave-wjdr krb5kdc[13210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375788, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.246.195@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM
I20260925 22:36:28.592252 13978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:28.577532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.195:53283 (local address 127.6.246.253:40799)
0925 22:36:28.577664 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:28.577668 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:28.578777 (+ 1109us) server_negotiation.cc:407] Connection header received
0925 22:36:28.580053 (+ 1276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:28.580057 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:28.580112 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:28.580166 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:28.581841 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:28.582358 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:28.584023 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:28.584261 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:28.584370 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:28.587783 (+ 3413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:28.587788 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:28.587790 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:28.587827 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:28.589922 (+ 2095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:28.590538 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:28.590542 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:28.590544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:28.590639 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:28.591032 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:28.591035 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:28.591037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:28.591259 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:28.591330 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:28.591974 (+ 644us) server_negotiation.cc:299] Negotiation successful
0925 22:36:28.592087 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 22:36:28.593163 13294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:28.577506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.195:43261 (local address 127.6.246.254:44671)
0925 22:36:28.577743 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:28.577750 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:28.578778 (+ 1028us) server_negotiation.cc:407] Connection header received
0925 22:36:28.580575 (+ 1797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:28.580579 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:28.580624 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:28.580709 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:28.581825 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:28.582626 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:28.584070 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:28.584418 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:28.584537 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:28.589241 (+ 4704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:28.589246 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:28.589250 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:28.589280 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:28.591154 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:28.591624 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:28.591627 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:28.591629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:28.591706 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:28.592004 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:28.592007 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:28.592009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:28.592193 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:28.592257 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:28.592629 (+ 372us) server_negotiation.cc:299] Negotiation successful
0925 22:36:28.592735 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137}
I20260925 22:36:28.593240 14497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:28.577955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.253:40799 (local address 127.6.246.195:53283)
0925 22:36:28.578631 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:28.578689 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:28.579824 (+ 1135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:28.580567 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:28.580576 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:28.581117 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:28.581682 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:28.581696 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:28.582506 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:28.582508 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:28.583905 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:28.583913 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:28.584367 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:28.584370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:28.584584 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:28.585202 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:28.585222 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:28.587617 (+ 2395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:36:28.590061 (+ 2444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:28.590068 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:28.590083 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:28.590416 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:28.590752 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:28.590754 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:28.590756 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:28.590926 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:28.591409 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:36:28.591414 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:36:28.591692 (+ 278us) client_negotiation.cc:769] Sending connection context
0925 22:36:28.592440 (+ 748us) client_negotiation.cc:240] Negotiation successful
0925 22:36:28.592592 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"mutex_wait_us":144,"spinlock_wait_cycles":14976,"thread_start_us":159,"threads_started":1}
I20260925 22:36:28.593492 14498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:28.579045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.252:46735 (local address 127.6.246.195:33163)
0925 22:36:28.579307 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:28.579320 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:28.579826 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:28.580567 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:28.580576 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:28.581117 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:28.581682 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:28.581696 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:28.583801 (+ 2105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:28.583804 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:28.584645 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:28.584657 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:28.585185 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:28.585188 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:28.585286 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:28.585775 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:28.585800 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:28.587596 (+ 1796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:36:28.590664 (+ 3068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:28.590670 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:28.590673 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:28.590897 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:28.591210 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:28.591212 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:28.591215 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:28.591300 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:28.591748 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:36:28.591751 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:36:28.591821 (+ 70us) client_negotiation.cc:769] Sending connection context
0925 22:36:28.591936 (+ 115us) client_negotiation.cc:240] Negotiation successful
0925 22:36:28.592226 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":81,"spinlock_wait_cycles":6272,"thread_start_us":94,"threads_started":1}
I20260925 22:36:28.593717 14496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:28.577955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:44671 (local address 127.6.246.195:43261)
0925 22:36:28.578631 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:28.578689 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:28.580390 (+ 1701us) spinlock_profiling.cc:243] Waited 912 us on lock 0x3558ede8. stack: 00007f4e69c8ac78 00007f4e69c8b1a9 00007f4e6984d1d7 00000000005984a6 00000000005984f9 000000000059c488 00007f4e69c872ab 00007f4e69c86ad0 00007f4e72a5e0a5 00007f4e72a5e58c 00007f4e723ed043 00007f4e6e2df267 00007f4e6e2dd819 00007f4e6e31e832 00007f4e6e31f68c 00007f4e6e3350d3
0925 22:36:28.580405 (+ 15us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:28.580725 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:28.580728 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:28.581116 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:28.581681 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:28.581694 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:28.582772 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:28.582776 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:28.583933 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:28.583943 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:28.586818 (+ 2875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:28.586824 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:28.586926 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:28.587589 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:28.587603 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:28.589042 (+ 1439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:36:28.591278 (+ 2236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:28.591281 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:28.591283 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:28.591517 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:28.591824 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:28.591827 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:28.591829 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:28.591911 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:28.592346 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:36:28.592350 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:36:28.592438 (+ 88us) client_negotiation.cc:769] Sending connection context
0925 22:36:28.593383 (+ 945us) client_negotiation.cc:240] Negotiation successful
0925 22:36:28.593510 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"mutex_wait_us":209,"spinlock_wait_cycles":2136960,"thread_start_us":137,"threads_started":1}
I20260925 22:36:28.594610 14493 heartbeater.cc:347] Connected to a master server at 127.6.246.254:44671
I20260925 22:36:28.594848 14493 heartbeater.cc:464] Registering TS with master...
I20260925 22:36:28.594988 14224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:28.579114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.195:33163 (local address 127.6.246.252:46735)
0925 22:36:28.579239 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:28.579243 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:28.579373 (+ 130us) server_negotiation.cc:407] Connection header received
0925 22:36:28.580053 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:28.580057 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:28.580114 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:28.580166 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:28.581856 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:28.582627 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:28.584772 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:28.585062 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:28.585186 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:28.588697 (+ 3511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:28.588702 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:28.588706 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:28.588733 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:28.590535 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:28.591032 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:28.591035 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:28.591037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:28.591113 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:28.591415 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:28.591418 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:28.591420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:28.591614 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:28.591707 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:28.594715 (+ 3008us) server_negotiation.cc:299] Negotiation successful
0925 22:36:28.594842 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 22:36:28.595327 14493 heartbeater.cc:511] Master 127.6.246.254:44671 requested a full tablet report, sending...
I20260925 22:36:28.595570 14492 heartbeater.cc:347] Connected to a master server at 127.6.246.252:46735
I20260925 22:36:28.595634 14492 heartbeater.cc:464] Registering TS with master...
I20260925 22:36:28.595770 14492 heartbeater.cc:511] Master 127.6.246.252:46735 requested a full tablet report, sending...
I20260925 22:36:28.594616 14494 heartbeater.cc:347] Connected to a master server at 127.6.246.253:40799
I20260925 22:36:28.595884 14494 heartbeater.cc:464] Registering TS with master...
I20260925 22:36:28.596009 14494 heartbeater.cc:511] Master 127.6.246.253:40799 requested a full tablet report, sending...
I20260925 22:36:28.596411 13255 ts_manager.cc:194] Re-registered known tserver with Master: d941f16284f0429381d37c34d0eaae2a (127.6.246.195:44879)
I20260925 22:36:28.596537 14034 ts_manager.cc:194] Registered new tserver with Master: d941f16284f0429381d37c34d0eaae2a (127.6.246.195:44879)
I20260925 22:36:28.596894 13856 ts_manager.cc:194] Registered new tserver with Master: d941f16284f0429381d37c34d0eaae2a (127.6.246.195:44879)
I20260925 22:36:28.597110 13255 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.246.195@KRBTEST.COM'} at 127.6.246.195:43261
I20260925 22:36:28.602274 13978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:28.587455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58002 (local address 127.6.246.253:40799)
0925 22:36:28.592404 (+ 4949us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:28.592407 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:28.594078 (+ 1671us) server_negotiation.cc:407] Connection header received
0925 22:36:28.594225 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:28.594229 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:28.594271 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:28.594341 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:28.595123 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:28.595679 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:28.596724 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:28.597052 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:28.597123 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:28.598716 (+ 1593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:28.598720 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:28.598724 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:28.598751 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:28.600438 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:28.600996 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:28.600999 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:28.601001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:28.601078 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:28.601386 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:28.601392 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:28.601397 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:28.601599 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:28.601726 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:28.601978 (+ 252us) server_negotiation.cc:299] Negotiation successful
0925 22:36:28.602084 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4895,"thread_start_us":96,"threads_started":1}
I20260925 22:36:28.602339 14503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:28.586950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53096 (local address 127.6.246.254:44671)
0925 22:36:28.587937 (+ 987us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:28.587953 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:28.588339 (+ 386us) server_negotiation.cc:407] Connection header received
0925 22:36:28.588382 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:28.588386 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:28.588437 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:28.588549 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:28.589668 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:28.590594 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:28.591827 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:28.592169 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:28.592259 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:28.598384 (+ 6125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:28.598388 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:28.598391 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:28.598417 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:28.600606 (+ 2189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:28.601130 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:28.601133 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:28.601135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:28.601220 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:28.601503 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:28.601506 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:28.601508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:28.601698 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:28.601819 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:28.602088 (+ 269us) server_negotiation.cc:299] Negotiation successful
0925 22:36:28.602203 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":917,"thread_start_us":62,"threads_started":1}
I20260925 22:36:28.606122 14508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:28.588492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35404 (local address 127.6.246.252:46735)
0925 22:36:28.593063 (+ 4571us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:28.593068 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:28.598349 (+ 5281us) server_negotiation.cc:407] Connection header received
0925 22:36:28.598387 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:28.598390 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:28.598433 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:28.598510 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:28.599277 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:28.599781 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:28.600553 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:28.600908 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:28.601010 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:28.602952 (+ 1942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:28.602955 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:28.602958 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:28.602982 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:28.604669 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:28.605141 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:28.605143 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:28.605145 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:28.605213 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:28.605477 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:28.605479 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:28.605480 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:28.605628 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:28.605719 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:28.605922 (+ 203us) server_negotiation.cc:299] Negotiation successful
0925 22:36:28.606008 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4462,"thread_start_us":94,"threads_started":1}
I20260925 22:36:28.605239 13255 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53096:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:36:28.608731 13255 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:36:28.611394 14381 sys_catalog.cc:455] T 00000000000000000000000000000000 P f241ea7630914735a806c4b2935b467d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f241ea7630914735a806c4b2935b467d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f241ea7630914735a806c4b2935b467d" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 44671 } } peers { permanent_uuid: "d481871900894bbdbed9d74795df7ece" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 40799 } attrs { promote: false } } peers { permanent_uuid: "3bf1e94e512547acba5b020a5eab47f5" member_type: NON_VOTER last_known_addr { host: "127.6.246.252" port: 46735 } attrs { promote: true } } }
I20260925 22:36:28.611537 14381 sys_catalog.cc:458] T 00000000000000000000000000000000 P f241ea7630914735a806c4b2935b467d [sys.catalog]: This master's current role is: LEADER
I20260925 22:36:28.614733 14381 raft_consensus.cc:1050] T 00000000000000000000000000000000 P f241ea7630914735a806c4b2935b467d: attempting to promote NON_VOTER 3bf1e94e512547acba5b020a5eab47f5 to VOTER
I20260925 22:36:28.615041 14381 consensus_queue.cc:237] T 00000000000000000000000000000000 P f241ea7630914735a806c4b2935b467d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f241ea7630914735a806c4b2935b467d" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 44671 } } peers { permanent_uuid: "d481871900894bbdbed9d74795df7ece" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 40799 } attrs { promote: false } } peers { permanent_uuid: "3bf1e94e512547acba5b020a5eab47f5" member_type: VOTER last_known_addr { host: "127.6.246.252" port: 46735 } attrs { promote: false } }
I20260925 22:36:28.616119 13866 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece [term 1 FOLLOWER]: Refusing update from remote peer f241ea7630914735a806c4b2935b467d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260925 22:36:28.616531 14510 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f241ea7630914735a806c4b2935b467d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d481871900894bbdbed9d74795df7ece" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 40799 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260925 22:36:28.617441 14048 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3bf1e94e512547acba5b020a5eab47f5 [term 1 LEARNER]: Refusing update from remote peer f241ea7630914735a806c4b2935b467d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260925 22:36:28.617962 14510 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f241ea7630914735a806c4b2935b467d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bf1e94e512547acba5b020a5eab47f5" member_type: VOTER last_known_addr { host: "127.6.246.252" port: 46735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260925 22:36:28.618553 14511 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f241ea7630914735a806c4b2935b467d [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 3bf1e94e512547acba5b020a5eab47f5 (127.6.246.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f241ea7630914735a806c4b2935b467d" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 44671 } } peers { permanent_uuid: "d481871900894bbdbed9d74795df7ece" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 40799 } attrs { promote: false } } peers { permanent_uuid: "3bf1e94e512547acba5b020a5eab47f5" member_type: VOTER last_known_addr { host: "127.6.246.252" port: 46735 } attrs { promote: false } } }
I20260925 22:36:28.619005 13866 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 3bf1e94e512547acba5b020a5eab47f5 (127.6.246.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f241ea7630914735a806c4b2935b467d" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 44671 } } peers { permanent_uuid: "d481871900894bbdbed9d74795df7ece" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 40799 } attrs { promote: false } } peers { permanent_uuid: "3bf1e94e512547acba5b020a5eab47f5" member_type: VOTER last_known_addr { host: "127.6.246.252" port: 46735 } attrs { promote: false } } }
I20260925 22:36:28.620057 14510 sys_catalog.cc:455] T 00000000000000000000000000000000 P f241ea7630914735a806c4b2935b467d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f241ea7630914735a806c4b2935b467d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f241ea7630914735a806c4b2935b467d" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 44671 } } peers { permanent_uuid: "d481871900894bbdbed9d74795df7ece" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 40799 } attrs { promote: false } } peers { permanent_uuid: "3bf1e94e512547acba5b020a5eab47f5" member_type: VOTER last_known_addr { host: "127.6.246.252" port: 46735 } attrs { promote: false } } }
I20260925 22:36:28.620157 14510 sys_catalog.cc:458] T 00000000000000000000000000000000 P f241ea7630914735a806c4b2935b467d [sys.catalog]: This master's current role is: LEADER
I20260925 22:36:28.620100 14511 sys_catalog.cc:455] T 00000000000000000000000000000000 P f241ea7630914735a806c4b2935b467d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f241ea7630914735a806c4b2935b467d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f241ea7630914735a806c4b2935b467d" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 44671 } } peers { permanent_uuid: "d481871900894bbdbed9d74795df7ece" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 40799 } attrs { promote: false } } peers { permanent_uuid: "3bf1e94e512547acba5b020a5eab47f5" member_type: VOTER last_known_addr { host: "127.6.246.252" port: 46735 } attrs { promote: false } } }
I20260925 22:36:28.620229 14511 sys_catalog.cc:458] T 00000000000000000000000000000000 P f241ea7630914735a806c4b2935b467d [sys.catalog]: This master's current role is: LEADER
I20260925 22:36:28.620266 14512 sys_catalog.cc:455] T 00000000000000000000000000000000 P f241ea7630914735a806c4b2935b467d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f241ea7630914735a806c4b2935b467d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f241ea7630914735a806c4b2935b467d" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 44671 } } peers { permanent_uuid: "d481871900894bbdbed9d74795df7ece" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 40799 } attrs { promote: false } } peers { permanent_uuid: "3bf1e94e512547acba5b020a5eab47f5" member_type: VOTER last_known_addr { host: "127.6.246.252" port: 46735 } attrs { promote: false } } }
I20260925 22:36:28.620363 14512 sys_catalog.cc:458] T 00000000000000000000000000000000 P f241ea7630914735a806c4b2935b467d [sys.catalog]: This master's current role is: LEADER
I20260925 22:36:28.620383 14048 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3bf1e94e512547acba5b020a5eab47f5 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 3bf1e94e512547acba5b020a5eab47f5 (127.6.246.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f241ea7630914735a806c4b2935b467d" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 44671 } } peers { permanent_uuid: "d481871900894bbdbed9d74795df7ece" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 40799 } attrs { promote: false } } peers { permanent_uuid: "3bf1e94e512547acba5b020a5eab47f5" member_type: VOTER last_known_addr { host: "127.6.246.252" port: 46735 } attrs { promote: false } } }
I20260925 22:36:28.621619 14520 sys_catalog.cc:455] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f241ea7630914735a806c4b2935b467d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f241ea7630914735a806c4b2935b467d" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 44671 } } peers { permanent_uuid: "d481871900894bbdbed9d74795df7ece" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 40799 } attrs { promote: false } } peers { permanent_uuid: "3bf1e94e512547acba5b020a5eab47f5" member_type: VOTER last_known_addr { host: "127.6.246.252" port: 46735 } attrs { promote: false } } }
I20260925 22:36:28.621742 14520 sys_catalog.cc:458] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:36:28.622121 14077 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bf1e94e512547acba5b020a5eab47f5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f241ea7630914735a806c4b2935b467d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f241ea7630914735a806c4b2935b467d" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 44671 } } peers { permanent_uuid: "d481871900894bbdbed9d74795df7ece" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 40799 } attrs { promote: false } } peers { permanent_uuid: "3bf1e94e512547acba5b020a5eab47f5" member_type: VOTER last_known_addr { host: "127.6.246.252" port: 46735 } attrs { promote: false } } }
I20260925 22:36:28.622231 14077 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bf1e94e512547acba5b020a5eab47f5 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:36:28.628437 14382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:28.622899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.195:44879 (local address 127.0.0.1:46762)
0925 22:36:28.623456 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:28.623472 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:28.623555 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:28.625085 (+ 1530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:28.625087 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:28.625102 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:36:28.625168 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:28.625173 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:28.626141 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:28.626144 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:28.626984 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:28.626992 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:28.628020 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:28.628023 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:28.628107 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:28.628130 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 22:36:28.628212 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 22:36:28.628282 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"mutex_wait_us":3,"thread_start_us":88,"threads_started":1}
I20260925 22:36:28.628484 14523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:28.623061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46762 (local address 127.6.246.195:44879)
0925 22:36:28.624640 (+ 1579us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:28.624648 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:28.624665 (+ 17us) server_negotiation.cc:407] Connection header received
0925 22:36:28.624722 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:28.624725 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:28.624935 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:28.625113 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:36:28.625258 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:28.626027 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:28.627124 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:28.627906 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:28.628003 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:28.628134 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:28.628237 (+ 103us) server_negotiation.cc:299] Negotiation successful
0925 22:36:28.628325 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1501,"thread_start_us":101,"threads_started":1}
I20260925 22:36:28.629177 14524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:28.623435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39642 (local address 127.6.246.194:36135)
0925 22:36:28.624310 (+ 875us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:28.624324 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:28.624348 (+ 24us) server_negotiation.cc:407] Connection header received
0925 22:36:28.624441 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:28.624446 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:28.624677 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:28.624842 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:36:28.625051 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:28.626225 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:28.627819 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:28.628664 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:28.628763 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:28.628911 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:28.628970 (+ 59us) server_negotiation.cc:299] Negotiation successful
0925 22:36:28.629050 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":770,"thread_start_us":97,"threads_started":1}
I20260925 22:36:28.629272 14521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:28.622859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.193:42341 (local address 127.0.0.1:33774)
0925 22:36:28.624023 (+ 1164us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:28.624038 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:28.624261 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:28.624939 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:28.624942 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:28.624961 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:36:28.625061 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:28.625068 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:28.626493 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:28.626497 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:28.627735 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:28.627745 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:28.628872 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:28.628875 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:28.628983 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:28.629005 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 22:36:28.629054 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 22:36:28.629115 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"mutex_wait_us":30}
I20260925 22:36:28.629123 14525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:28.623208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.194:36135 (local address 127.0.0.1:39642)
0925 22:36:28.623841 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:28.623869 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:28.623955 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:28.624832 (+ 877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:28.624835 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:28.624852 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:36:28.624952 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:28.624959 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:28.626354 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:28.626357 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:28.627716 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:28.627728 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:28.628761 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:28.628764 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:28.628847 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:28.628867 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 22:36:28.628938 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 22:36:28.629007 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"mutex_wait_us":490,"thread_start_us":84,"threads_started":1}
I20260925 22:36:28.629643 14522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:28.623031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33774 (local address 127.6.246.193:42341)
0925 22:36:28.624526 (+ 1495us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:28.624537 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:28.624557 (+ 20us) server_negotiation.cc:407] Connection header received
0925 22:36:28.624622 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:28.624627 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:28.624785 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:28.625210 (+ 425us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:36:28.625343 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:28.626380 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:28.627880 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:28.628716 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:28.629190 (+ 474us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:28.629331 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:28.629377 (+ 46us) server_negotiation.cc:299] Negotiation successful
0925 22:36:28.629458 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1380,"thread_start_us":97,"threads_started":1}
I20260925 22:36:28.631587 14426 tablet_service.cc:1467] Processing CreateTablet for tablet 924ee57af16940baa9549901d0ddefc8 (DEFAULT_TABLE table=test-table [id=fe0965bbd1f9411aa25c98c8025232f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:36:28.631999 14151 tablet_service.cc:1467] Processing CreateTablet for tablet 924ee57af16940baa9549901d0ddefc8 (DEFAULT_TABLE table=test-table [id=fe0965bbd1f9411aa25c98c8025232f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:36:28.632521 14426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 924ee57af16940baa9549901d0ddefc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:36:28.632894 14151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 924ee57af16940baa9549901d0ddefc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:36:28.638156 14288 tablet_service.cc:1467] Processing CreateTablet for tablet 924ee57af16940baa9549901d0ddefc8 (DEFAULT_TABLE table=test-table [id=fe0965bbd1f9411aa25c98c8025232f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:36:28.639130 14288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 924ee57af16940baa9549901d0ddefc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:36:28.640128 14526 tablet_bootstrap.cc:492] T 924ee57af16940baa9549901d0ddefc8 P 67b78fd0bb194aa0b4b9cefa890e343f: Bootstrap starting.
I20260925 22:36:28.640744 14527 tablet_bootstrap.cc:492] T 924ee57af16940baa9549901d0ddefc8 P d941f16284f0429381d37c34d0eaae2a: Bootstrap starting.
I20260925 22:36:28.642809 14527 tablet_bootstrap.cc:654] T 924ee57af16940baa9549901d0ddefc8 P d941f16284f0429381d37c34d0eaae2a: Neither blocks nor log segments found. Creating new log.
I20260925 22:36:28.643550 14527 log.cc:824] T 924ee57af16940baa9549901d0ddefc8 P d941f16284f0429381d37c34d0eaae2a: Log is configured to *not* fsync() on all Append() calls
I20260925 22:36:28.645430 14526 tablet_bootstrap.cc:654] T 924ee57af16940baa9549901d0ddefc8 P 67b78fd0bb194aa0b4b9cefa890e343f: Neither blocks nor log segments found. Creating new log.
I20260925 22:36:28.645769 14529 tablet_bootstrap.cc:492] T 924ee57af16940baa9549901d0ddefc8 P 8d3bf93a4c6844cf9013e229796c55d1: Bootstrap starting.
I20260925 22:36:28.646257 14526 log.cc:824] T 924ee57af16940baa9549901d0ddefc8 P 67b78fd0bb194aa0b4b9cefa890e343f: Log is configured to *not* fsync() on all Append() calls
I20260925 22:36:28.647586 14529 tablet_bootstrap.cc:654] T 924ee57af16940baa9549901d0ddefc8 P 8d3bf93a4c6844cf9013e229796c55d1: Neither blocks nor log segments found. Creating new log.
I20260925 22:36:28.647804 14527 tablet_bootstrap.cc:492] T 924ee57af16940baa9549901d0ddefc8 P d941f16284f0429381d37c34d0eaae2a: No bootstrap required, opened a new log
I20260925 22:36:28.647969 14527 ts_tablet_manager.cc:1397] T 924ee57af16940baa9549901d0ddefc8 P d941f16284f0429381d37c34d0eaae2a: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260925 22:36:28.649147 14529 log.cc:824] T 924ee57af16940baa9549901d0ddefc8 P 8d3bf93a4c6844cf9013e229796c55d1: Log is configured to *not* fsync() on all Append() calls
I20260925 22:36:28.649457 14526 tablet_bootstrap.cc:492] T 924ee57af16940baa9549901d0ddefc8 P 67b78fd0bb194aa0b4b9cefa890e343f: No bootstrap required, opened a new log
I20260925 22:36:28.649611 14526 ts_tablet_manager.cc:1397] T 924ee57af16940baa9549901d0ddefc8 P 67b78fd0bb194aa0b4b9cefa890e343f: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20260925 22:36:28.650669 14529 tablet_bootstrap.cc:492] T 924ee57af16940baa9549901d0ddefc8 P 8d3bf93a4c6844cf9013e229796c55d1: No bootstrap required, opened a new log
I20260925 22:36:28.650846 14529 ts_tablet_manager.cc:1397] T 924ee57af16940baa9549901d0ddefc8 P 8d3bf93a4c6844cf9013e229796c55d1: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 22:36:28.654315 14529 raft_consensus.cc:348] T 924ee57af16940baa9549901d0ddefc8 P 8d3bf93a4c6844cf9013e229796c55d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d3bf93a4c6844cf9013e229796c55d1" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 36135 } } peers { permanent_uuid: "d941f16284f0429381d37c34d0eaae2a" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 44879 } } peers { permanent_uuid: "67b78fd0bb194aa0b4b9cefa890e343f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 42341 } }
I20260925 22:36:28.654523 14529 raft_consensus.cc:374] T 924ee57af16940baa9549901d0ddefc8 P 8d3bf93a4c6844cf9013e229796c55d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:36:28.654574 14529 raft_consensus.cc:729] T 924ee57af16940baa9549901d0ddefc8 P 8d3bf93a4c6844cf9013e229796c55d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d3bf93a4c6844cf9013e229796c55d1, State: Initialized, Role: FOLLOWER
I20260925 22:36:28.654990 14529 consensus_queue.cc:260] T 924ee57af16940baa9549901d0ddefc8 P 8d3bf93a4c6844cf9013e229796c55d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d3bf93a4c6844cf9013e229796c55d1" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 36135 } } peers { permanent_uuid: "d941f16284f0429381d37c34d0eaae2a" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 44879 } } peers { permanent_uuid: "67b78fd0bb194aa0b4b9cefa890e343f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 42341 } }
I20260925 22:36:28.655179 14526 raft_consensus.cc:348] T 924ee57af16940baa9549901d0ddefc8 P 67b78fd0bb194aa0b4b9cefa890e343f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d3bf93a4c6844cf9013e229796c55d1" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 36135 } } peers { permanent_uuid: "d941f16284f0429381d37c34d0eaae2a" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 44879 } } peers { permanent_uuid: "67b78fd0bb194aa0b4b9cefa890e343f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 42341 } }
I20260925 22:36:28.655468 14526 raft_consensus.cc:374] T 924ee57af16940baa9549901d0ddefc8 P 67b78fd0bb194aa0b4b9cefa890e343f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:36:28.655539 14526 raft_consensus.cc:729] T 924ee57af16940baa9549901d0ddefc8 P 67b78fd0bb194aa0b4b9cefa890e343f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67b78fd0bb194aa0b4b9cefa890e343f, State: Initialized, Role: FOLLOWER
I20260925 22:36:28.655947 14526 consensus_queue.cc:260] T 924ee57af16940baa9549901d0ddefc8 P 67b78fd0bb194aa0b4b9cefa890e343f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d3bf93a4c6844cf9013e229796c55d1" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 36135 } } peers { permanent_uuid: "d941f16284f0429381d37c34d0eaae2a" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 44879 } } peers { permanent_uuid: "67b78fd0bb194aa0b4b9cefa890e343f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 42341 } }
I20260925 22:36:28.656589 14355 heartbeater.cc:503] Master 127.6.246.254:44671 was elected leader, sending a full tablet report...
I20260925 22:36:28.656433 14527 raft_consensus.cc:348] T 924ee57af16940baa9549901d0ddefc8 P d941f16284f0429381d37c34d0eaae2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d3bf93a4c6844cf9013e229796c55d1" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 36135 } } peers { permanent_uuid: "d941f16284f0429381d37c34d0eaae2a" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 44879 } } peers { permanent_uuid: "67b78fd0bb194aa0b4b9cefa890e343f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 42341 } }
I20260925 22:36:28.656689 14527 raft_consensus.cc:374] T 924ee57af16940baa9549901d0ddefc8 P d941f16284f0429381d37c34d0eaae2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:36:28.656740 14527 raft_consensus.cc:729] T 924ee57af16940baa9549901d0ddefc8 P d941f16284f0429381d37c34d0eaae2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d941f16284f0429381d37c34d0eaae2a, State: Initialized, Role: FOLLOWER
I20260925 22:36:28.656924 14526 ts_tablet_manager.cc:1428] T 924ee57af16940baa9549901d0ddefc8 P 67b78fd0bb194aa0b4b9cefa890e343f: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260925 22:36:28.657299 14218 heartbeater.cc:503] Master 127.6.246.254:44671 was elected leader, sending a full tablet report...
I20260925 22:36:28.657164 14527 consensus_queue.cc:260] T 924ee57af16940baa9549901d0ddefc8 P d941f16284f0429381d37c34d0eaae2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d3bf93a4c6844cf9013e229796c55d1" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 36135 } } peers { permanent_uuid: "d941f16284f0429381d37c34d0eaae2a" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 44879 } } peers { permanent_uuid: "67b78fd0bb194aa0b4b9cefa890e343f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 42341 } }
I20260925 22:36:28.658659 14527 ts_tablet_manager.cc:1428] T 924ee57af16940baa9549901d0ddefc8 P d941f16284f0429381d37c34d0eaae2a: Time spent starting tablet: real 0.011s user 0.000s sys 0.006s
I20260925 22:36:28.659103 14493 heartbeater.cc:503] Master 127.6.246.254:44671 was elected leader, sending a full tablet report...
I20260925 22:36:28.660028 14529 ts_tablet_manager.cc:1428] T 924ee57af16940baa9549901d0ddefc8 P 8d3bf93a4c6844cf9013e229796c55d1: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
W20260925 22:36:28.691926 14220 tablet.cc:2369] T 924ee57af16940baa9549901d0ddefc8 P 67b78fd0bb194aa0b4b9cefa890e343f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:36:28.717521 14532 raft_consensus.cc:482] T 924ee57af16940baa9549901d0ddefc8 P 8d3bf93a4c6844cf9013e229796c55d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:36:28.717721 14532 raft_consensus.cc:504] T 924ee57af16940baa9549901d0ddefc8 P 8d3bf93a4c6844cf9013e229796c55d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d3bf93a4c6844cf9013e229796c55d1" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 36135 } } peers { permanent_uuid: "d941f16284f0429381d37c34d0eaae2a" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 44879 } } peers { permanent_uuid: "67b78fd0bb194aa0b4b9cefa890e343f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 42341 } }
I20260925 22:36:28.719059 14532 leader_election.cc:288] T 924ee57af16940baa9549901d0ddefc8 P 8d3bf93a4c6844cf9013e229796c55d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d941f16284f0429381d37c34d0eaae2a (127.6.246.195:44879), 67b78fd0bb194aa0b4b9cefa890e343f (127.6.246.193:42341)
I20260925 22:36:28.722791 14523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:28.719609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.194:48353 (local address 127.6.246.195:44879)
0925 22:36:28.719733 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:28.719736 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:28.719766 (+ 30us) server_negotiation.cc:407] Connection header received
0925 22:36:28.719813 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:28.719816 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:28.719858 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:28.719943 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:36:28.720227 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:28.720775 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:28.721809 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:28.722194 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:28.722316 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:28.722366 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:28.722588 (+ 222us) server_negotiation.cc:299] Negotiation successful
0925 22:36:28.722662 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 22:36:28.722800 14359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:28.719237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.193:42341 (local address 127.6.246.194:55507)
0925 22:36:28.719429 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:28.719448 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:28.719535 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:28.719800 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:28.719802 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:28.719819 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:36:28.719947 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:28.719953 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:28.720794 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:28.720800 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:28.721660 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:28.721669 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:28.722329 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:28.722332 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:28.722487 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:28.722505 (+ 18us) client_negotiation.cc:769] Sending connection context
0925 22:36:28.722600 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 22:36:28.722667 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":88}
I20260925 22:36:28.722806 14522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:28.719355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.194:55507 (local address 127.6.246.193:42341)
0925 22:36:28.719503 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:28.719508 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:28.719524 (+ 16us) server_negotiation.cc:407] Connection header received
0925 22:36:28.719631 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:28.719634 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:28.719698 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:28.719773 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:36:28.720053 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:28.720694 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:28.721830 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:28.722204 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:28.722330 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:28.722380 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:28.722623 (+ 243us) server_negotiation.cc:299] Negotiation successful
0925 22:36:28.722704 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 22:36:28.722801 14360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:28.719237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.195:44879 (local address 127.6.246.194:48353)
0925 22:36:28.719550 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:28.719564 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:28.719656 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:28.719964 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:28.719966 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:28.719979 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:36:28.720117 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:28.720124 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:28.720885 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:28.720888 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:28.721680 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:28.721688 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:28.722308 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:28.722312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:28.722459 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:28.722483 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 22:36:28.722543 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 22:36:28.722626 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"mutex_wait_us":208}
I20260925 22:36:28.723423 14446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "924ee57af16940baa9549901d0ddefc8" candidate_uuid: "8d3bf93a4c6844cf9013e229796c55d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d941f16284f0429381d37c34d0eaae2a" is_pre_election: true
I20260925 22:36:28.723425 14171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "924ee57af16940baa9549901d0ddefc8" candidate_uuid: "8d3bf93a4c6844cf9013e229796c55d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b78fd0bb194aa0b4b9cefa890e343f" is_pre_election: true
I20260925 22:36:28.723688 14446 raft_consensus.cc:2454] T 924ee57af16940baa9549901d0ddefc8 P d941f16284f0429381d37c34d0eaae2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d3bf93a4c6844cf9013e229796c55d1 in term 0.
I20260925 22:36:28.723719 14171 raft_consensus.cc:2454] T 924ee57af16940baa9549901d0ddefc8 P 67b78fd0bb194aa0b4b9cefa890e343f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d3bf93a4c6844cf9013e229796c55d1 in term 0.
I20260925 22:36:28.724115 14244 leader_election.cc:302] T 924ee57af16940baa9549901d0ddefc8 P 8d3bf93a4c6844cf9013e229796c55d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d3bf93a4c6844cf9013e229796c55d1, d941f16284f0429381d37c34d0eaae2a; no voters:
I20260925 22:36:28.724381 14532 raft_consensus.cc:2781] T 924ee57af16940baa9549901d0ddefc8 P 8d3bf93a4c6844cf9013e229796c55d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:36:28.724496 14532 raft_consensus.cc:482] T 924ee57af16940baa9549901d0ddefc8 P 8d3bf93a4c6844cf9013e229796c55d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:36:28.724552 14532 raft_consensus.cc:3037] T 924ee57af16940baa9549901d0ddefc8 P 8d3bf93a4c6844cf9013e229796c55d1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:36:28.725528 14532 raft_consensus.cc:504] T 924ee57af16940baa9549901d0ddefc8 P 8d3bf93a4c6844cf9013e229796c55d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d3bf93a4c6844cf9013e229796c55d1" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 36135 } } peers { permanent_uuid: "d941f16284f0429381d37c34d0eaae2a" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 44879 } } peers { permanent_uuid: "67b78fd0bb194aa0b4b9cefa890e343f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 42341 } }
I20260925 22:36:28.725857 14532 leader_election.cc:288] T 924ee57af16940baa9549901d0ddefc8 P 8d3bf93a4c6844cf9013e229796c55d1 [CANDIDATE]: Term 1 election: Requested vote from peers d941f16284f0429381d37c34d0eaae2a (127.6.246.195:44879), 67b78fd0bb194aa0b4b9cefa890e343f (127.6.246.193:42341)
I20260925 22:36:28.726135 14446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "924ee57af16940baa9549901d0ddefc8" candidate_uuid: "8d3bf93a4c6844cf9013e229796c55d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d941f16284f0429381d37c34d0eaae2a"
I20260925 22:36:28.726166 14171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "924ee57af16940baa9549901d0ddefc8" candidate_uuid: "8d3bf93a4c6844cf9013e229796c55d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b78fd0bb194aa0b4b9cefa890e343f"
I20260925 22:36:28.726284 14446 raft_consensus.cc:3037] T 924ee57af16940baa9549901d0ddefc8 P d941f16284f0429381d37c34d0eaae2a [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:36:28.726275 14171 raft_consensus.cc:3037] T 924ee57af16940baa9549901d0ddefc8 P 67b78fd0bb194aa0b4b9cefa890e343f [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:36:28.727228 14446 raft_consensus.cc:2454] T 924ee57af16940baa9549901d0ddefc8 P d941f16284f0429381d37c34d0eaae2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d3bf93a4c6844cf9013e229796c55d1 in term 1.
I20260925 22:36:28.727245 14171 raft_consensus.cc:2454] T 924ee57af16940baa9549901d0ddefc8 P 67b78fd0bb194aa0b4b9cefa890e343f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d3bf93a4c6844cf9013e229796c55d1 in term 1.
I20260925 22:36:28.727542 14243 leader_election.cc:302] T 924ee57af16940baa9549901d0ddefc8 P 8d3bf93a4c6844cf9013e229796c55d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b78fd0bb194aa0b4b9cefa890e343f, 8d3bf93a4c6844cf9013e229796c55d1; no voters:
I20260925 22:36:28.727701 14532 raft_consensus.cc:2781] T 924ee57af16940baa9549901d0ddefc8 P 8d3bf93a4c6844cf9013e229796c55d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:36:28.727932 14532 raft_consensus.cc:686] T 924ee57af16940baa9549901d0ddefc8 P 8d3bf93a4c6844cf9013e229796c55d1 [term 1 LEADER]: Becoming Leader. State: Replica: 8d3bf93a4c6844cf9013e229796c55d1, State: Running, Role: LEADER
I20260925 22:36:28.728259 14532 consensus_queue.cc:237] T 924ee57af16940baa9549901d0ddefc8 P 8d3bf93a4c6844cf9013e229796c55d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d3bf93a4c6844cf9013e229796c55d1" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 36135 } } peers { permanent_uuid: "d941f16284f0429381d37c34d0eaae2a" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 44879 } } peers { permanent_uuid: "67b78fd0bb194aa0b4b9cefa890e343f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 42341 } }
I20260925 22:36:28.731405 13255 catalog_manager.cc:5495] T 924ee57af16940baa9549901d0ddefc8 P 8d3bf93a4c6844cf9013e229796c55d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d3bf93a4c6844cf9013e229796c55d1 (127.6.246.194). New cstate: current_term: 1 leader_uuid: "8d3bf93a4c6844cf9013e229796c55d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d3bf93a4c6844cf9013e229796c55d1" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 36135 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d941f16284f0429381d37c34d0eaae2a" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 44879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67b78fd0bb194aa0b4b9cefa890e343f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 42341 } health_report { overall_health: UNKNOWN } } }
I20260925 22:36:28.750011 14524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:28.746522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39658 (local address 127.6.246.194:36135)
0925 22:36:28.746639 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:28.746643 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:28.746685 (+ 42us) server_negotiation.cc:407] Connection header received
0925 22:36:28.746835 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:28.746837 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:28.746902 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:28.746985 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:36:28.747198 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:28.747668 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:28.748675 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:28.748872 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:28.748950 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:28.749191 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:36:28.749621 (+ 430us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:36:28.749704 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:28.749778 (+ 74us) server_negotiation.cc:299] Negotiation successful
0925 22:36:28.749823 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 22:36:28.755597 14446 raft_consensus.cc:1261] T 924ee57af16940baa9549901d0ddefc8 P d941f16284f0429381d37c34d0eaae2a [term 1 FOLLOWER]: Refusing update from remote peer 8d3bf93a4c6844cf9013e229796c55d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:36:28.755661 14171 raft_consensus.cc:1261] T 924ee57af16940baa9549901d0ddefc8 P 67b78fd0bb194aa0b4b9cefa890e343f [term 1 FOLLOWER]: Refusing update from remote peer 8d3bf93a4c6844cf9013e229796c55d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:36:28.756317 14535 consensus_queue.cc:1035] T 924ee57af16940baa9549901d0ddefc8 P 8d3bf93a4c6844cf9013e229796c55d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d941f16284f0429381d37c34d0eaae2a" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 44879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:36:28.756500 14532 consensus_queue.cc:1035] T 924ee57af16940baa9549901d0ddefc8 P 8d3bf93a4c6844cf9013e229796c55d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b78fd0bb194aa0b4b9cefa890e343f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 42341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:36:28.766489 14541 mvcc.cc:205] Tried to move back new op lower bound from 7333379230736228352 to 7333379230633697280. Current Snapshot: MvccSnapshot[applied={T|T < 7333379230736228352}]
I20260925 22:36:28.768559 14538 mvcc.cc:205] Tried to move back new op lower bound from 7333379230736228352 to 7333379230633697280. Current Snapshot: MvccSnapshot[applied={T|T < 7333379230736228352 or (T in {7333379230736228352})}]
I20260925 22:36:28.772447 13255 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53096:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:36:28.772615 13255 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53096:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:36:28.776638 13255 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f241ea7630914735a806c4b2935b467d: Sending DeleteTablet for 3 replicas of tablet 924ee57af16940baa9549901d0ddefc8
I20260925 22:36:28.777496 14426 tablet_service.cc:1514] Processing DeleteTablet for tablet 924ee57af16940baa9549901d0ddefc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:36:28 UTC) from {username='oryx', principal='oryx/127.6.246.254@KRBTEST.COM'} at 127.0.0.1:46762
I20260925 22:36:28.777817 14151 tablet_service.cc:1514] Processing DeleteTablet for tablet 924ee57af16940baa9549901d0ddefc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:36:28 UTC) from {username='oryx', principal='oryx/127.6.246.254@KRBTEST.COM'} at 127.0.0.1:33774
I20260925 22:36:28.778388 14288 tablet_service.cc:1514] Processing DeleteTablet for tablet 924ee57af16940baa9549901d0ddefc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:36:28 UTC) from {username='oryx', principal='oryx/127.6.246.254@KRBTEST.COM'} at 127.0.0.1:39642
I20260925 22:36:28.779013 14539 mvcc.cc:205] Tried to move back new op lower bound from 7333379230736228352 to 7333379230633697280. Current Snapshot: MvccSnapshot[applied={T|T < 7333379230736228352}]
I20260925 22:36:28.779173 14547 tablet_replica.cc:330] T 924ee57af16940baa9549901d0ddefc8 P d941f16284f0429381d37c34d0eaae2a: stopping tablet replica
I20260925 22:36:28.779455 14547 raft_consensus.cc:2229] T 924ee57af16940baa9549901d0ddefc8 P d941f16284f0429381d37c34d0eaae2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:36:28.779704 14547 raft_consensus.cc:2258] T 924ee57af16940baa9549901d0ddefc8 P d941f16284f0429381d37c34d0eaae2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:36:28.780645 14549 tablet_replica.cc:330] T 924ee57af16940baa9549901d0ddefc8 P 8d3bf93a4c6844cf9013e229796c55d1: stopping tablet replica
I20260925 22:36:28.781050 14549 raft_consensus.cc:2229] T 924ee57af16940baa9549901d0ddefc8 P 8d3bf93a4c6844cf9013e229796c55d1 [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:36:28.781473 14549 raft_consensus.cc:2258] T 924ee57af16940baa9549901d0ddefc8 P 8d3bf93a4c6844cf9013e229796c55d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:36:28.781487 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 13223
I20260925 22:36:28.782235 14548 tablet_replica.cc:330] T 924ee57af16940baa9549901d0ddefc8 P 67b78fd0bb194aa0b4b9cefa890e343f: stopping tablet replica
I20260925 22:36:28.783653 14549 ts_tablet_manager.cc:1905] T 924ee57af16940baa9549901d0ddefc8 P 8d3bf93a4c6844cf9013e229796c55d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:36:28.784027 14548 raft_consensus.cc:2229] T 924ee57af16940baa9549901d0ddefc8 P 67b78fd0bb194aa0b4b9cefa890e343f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:36:28.784373 14548 raft_consensus.cc:2258] T 924ee57af16940baa9549901d0ddefc8 P 67b78fd0bb194aa0b4b9cefa890e343f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:36:28.787784 14549 ts_tablet_manager.cc:1918] T 924ee57af16940baa9549901d0ddefc8 P 8d3bf93a4c6844cf9013e229796c55d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:36:28.788537 14549 log.cc:1196] T 924ee57af16940baa9549901d0ddefc8 P 8d3bf93a4c6844cf9013e229796c55d1: Deleting WAL directory at /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-1/wal/wals/924ee57af16940baa9549901d0ddefc8
I20260925 22:36:28.789001 14549 ts_tablet_manager.cc:1939] T 924ee57af16940baa9549901d0ddefc8 P 8d3bf93a4c6844cf9013e229796c55d1: Deleting consensus metadata
I20260925 22:36:28.789727 13978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:28.779697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58006 (local address 127.6.246.253:40799)
0925 22:36:28.779868 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:28.779875 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:28.779897 (+ 22us) server_negotiation.cc:407] Connection header received
0925 22:36:28.779937 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:28.779939 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:28.779984 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:28.780123 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:28.781018 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:28.781777 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:28.783130 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:28.783535 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:28.783627 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:28.785724 (+ 2097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:28.785730 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:28.785736 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:28.785766 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:28.787606 (+ 1840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:28.788154 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:28.788161 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:28.788166 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:28.788250 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:28.788603 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:28.788609 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:28.788614 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:28.788862 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:28.788974 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:28.789351 (+ 377us) server_negotiation.cc:299] Negotiation successful
0925 22:36:28.789495 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260925 22:36:28.791474 14548 ts_tablet_manager.cc:1905] T 924ee57af16940baa9549901d0ddefc8 P 67b78fd0bb194aa0b4b9cefa890e343f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:36:28.794373 14548 ts_tablet_manager.cc:1918] T 924ee57af16940baa9549901d0ddefc8 P 67b78fd0bb194aa0b4b9cefa890e343f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:36:28.794530 14548 log.cc:1196] T 924ee57af16940baa9549901d0ddefc8 P 67b78fd0bb194aa0b4b9cefa890e343f: Deleting WAL directory at /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-0/wal/wals/924ee57af16940baa9549901d0ddefc8
I20260925 22:36:28.794936 14548 ts_tablet_manager.cc:1939] T 924ee57af16940baa9549901d0ddefc8 P 67b78fd0bb194aa0b4b9cefa890e343f: Deleting consensus metadata
W20260925 22:36:28.796362 14244 connection.cc:441] server connection from 127.0.0.1:39642 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39642 (request call id 1) could send its response
I20260925 22:36:28.797340 14547 ts_tablet_manager.cc:1905] T 924ee57af16940baa9549901d0ddefc8 P d941f16284f0429381d37c34d0eaae2a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 22:36:28.797857 14107 connection.cc:441] server connection from 127.0.0.1:33774 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33774 (request call id 1) could send its response
I20260925 22:36:28.799746 14508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:28.779729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35414 (local address 127.6.246.252:46735)
0925 22:36:28.779834 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:28.779839 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:28.790342 (+ 10503us) server_negotiation.cc:407] Connection header received
0925 22:36:28.790545 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:28.790548 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:28.790599 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:28.790652 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:28.791513 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:28.792282 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:28.793445 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:28.793816 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:28.793919 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:28.795831 (+ 1912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:28.795834 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:28.795837 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:28.795861 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:28.797565 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:28.798084 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:28.798088 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:28.798089 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:28.798169 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:28.798703 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:28.798707 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:28.798708 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:28.798956 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:28.799062 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:28.799457 (+ 395us) server_negotiation.cc:299] Negotiation successful
0925 22:36:28.799591 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 22:36:28.800266 14547 ts_tablet_manager.cc:1918] T 924ee57af16940baa9549901d0ddefc8 P d941f16284f0429381d37c34d0eaae2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:36:28.800382 14547 log.cc:1196] T 924ee57af16940baa9549901d0ddefc8 P d941f16284f0429381d37c34d0eaae2a: Deleting WAL directory at /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-2/wal/wals/924ee57af16940baa9549901d0ddefc8
I20260925 22:36:28.800681 14547 ts_tablet_manager.cc:1939] T 924ee57af16940baa9549901d0ddefc8 P d941f16284f0429381d37c34d0eaae2a: Deleting consensus metadata
W20260925 22:36:28.801635 14379 connection.cc:441] server connection from 127.0.0.1:46762 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46762 (request call id 1) could send its response
I20260925 22:36:28.810472 14508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:28.801165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35420 (local address 127.6.246.252:46735)
0925 22:36:28.801269 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:28.801273 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:28.801636 (+ 363us) server_negotiation.cc:407] Connection header received
0925 22:36:28.801781 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:28.801784 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:28.801827 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:28.801921 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:28.802896 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:28.803537 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:28.804914 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:28.805196 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:28.805301 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:28.807166 (+ 1865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:28.807169 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:28.807172 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:28.807195 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:28.808724 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:28.809219 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:28.809224 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:28.809226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:28.809304 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:28.809566 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:28.809570 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:28.809572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:28.809769 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:28.809887 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:28.810193 (+ 306us) server_negotiation.cc:299] Negotiation successful
0925 22:36:28.810293 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 22:36:28.813277 13978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:28.795414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58020 (local address 127.6.246.253:40799)
0925 22:36:28.795550 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:28.795554 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:28.804717 (+ 9163us) server_negotiation.cc:407] Connection header received
0925 22:36:28.804828 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:28.804831 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:28.804914 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:28.804992 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:28.805826 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:28.806310 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:28.807528 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:28.807919 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:28.807995 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:28.809823 (+ 1828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:28.809830 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:28.809835 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:28.809858 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:28.811713 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:28.812093 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:28.812097 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:28.812099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:28.812187 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:28.812423 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:28.812426 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:28.812428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:28.812696 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:28.812893 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:28.812994 (+ 101us) server_negotiation.cc:299] Negotiation successful
0925 22:36:28.813113 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 22:36:29.767858 14558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:29.767211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:44671 (local address 127.6.246.194:53297)
0925 22:36:29.767575 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:29.767725 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:44671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":119,"threads_started":1}
W20260925 22:36:29.768272 14355 heartbeater.cc:644] Failed to heartbeat to 127.6.246.254:44671 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.6.246.254:44671: connect: Connection refused (error 111)
I20260925 22:36:29.771607 14559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:29.771117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:44671 (local address 127.6.246.195:60943)
0925 22:36:29.771408 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:29.771499 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:44671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":100,"threads_started":1}
W20260925 22:36:29.771948 14493 heartbeater.cc:644] Failed to heartbeat to 127.6.246.254:44671 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.6.246.254:44671: connect: Connection refused (error 111)
I20260925 22:36:29.772104 14560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:29.771375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:44671 (local address 127.6.246.193:49533)
0925 22:36:29.771612 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:29.771971 (+ 359us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:44671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":85,"threads_started":1}
W20260925 22:36:29.772514 14218 heartbeater.cc:644] Failed to heartbeat to 127.6.246.254:44671 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.6.246.254:44671: connect: Connection refused (error 111)
I20260925 22:36:30.276793 14561 raft_consensus.cc:482] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f241ea7630914735a806c4b2935b467d)
I20260925 22:36:30.277148 14561 raft_consensus.cc:504] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f241ea7630914735a806c4b2935b467d" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 44671 } } peers { permanent_uuid: "d481871900894bbdbed9d74795df7ece" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 40799 } attrs { promote: false } } peers { permanent_uuid: "3bf1e94e512547acba5b020a5eab47f5" member_type: VOTER last_known_addr { host: "127.6.246.252" port: 46735 } attrs { promote: false } }
I20260925 22:36:30.278024 14562 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3bf1e94e512547acba5b020a5eab47f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f241ea7630914735a806c4b2935b467d)
I20260925 22:36:30.278338 14562 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3bf1e94e512547acba5b020a5eab47f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f241ea7630914735a806c4b2935b467d" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 44671 } } peers { permanent_uuid: "d481871900894bbdbed9d74795df7ece" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 40799 } attrs { promote: false } } peers { permanent_uuid: "3bf1e94e512547acba5b020a5eab47f5" member_type: VOTER last_known_addr { host: "127.6.246.252" port: 46735 } attrs { promote: false } }
I20260925 22:36:30.278708 14561 leader_election.cc:288] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f241ea7630914735a806c4b2935b467d (127.6.246.254:44671), 3bf1e94e512547acba5b020a5eab47f5 (127.6.246.252:46735)
I20260925 22:36:30.279222 14563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:30.278798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:44671 (local address 127.0.0.1:53568)
0925 22:36:30.279006 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:30.279099 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:44671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":69,"threads_started":1}
I20260925 22:36:30.279624 14562 leader_election.cc:288] T 00000000000000000000000000000000 P 3bf1e94e512547acba5b020a5eab47f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f241ea7630914735a806c4b2935b467d (127.6.246.254:44671), d481871900894bbdbed9d74795df7ece (127.6.246.253:40799)
W20260925 22:36:30.279991 13840 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.246.254:44671: connect: Connection refused (error 111)
I20260925 22:36:30.280275 14566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:30.279795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:44671 (local address 127.0.0.1:53580)
0925 22:36:30.280105 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:30.280175 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:44671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":66,"threads_started":1}
W20260925 22:36:30.280597 14022 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.246.254:44671: connect: Connection refused (error 111)
I20260925 22:36:30.281673 14566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:30.281418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:44671 (local address 127.0.0.1:53586)
0925 22:36:30.281546 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:30.281604 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:44671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260925 22:36:30.281714 14564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:30.281430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:44671 (local address 127.0.0.1:53600)
0925 22:36:30.281580 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:30.281645 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:44671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260925 22:36:30.281790 14022 leader_election.cc:334] T 00000000000000000000000000000000 P 3bf1e94e512547acba5b020a5eab47f5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f241ea7630914735a806c4b2935b467d (127.6.246.254:44671): Network error: Client connection negotiation failed: client connection to 127.6.246.254:44671: connect: Connection refused (error 111)
W20260925 22:36:30.281955 13840 leader_election.cc:334] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f241ea7630914735a806c4b2935b467d (127.6.246.254:44671): Network error: Client connection negotiation failed: client connection to 127.6.246.254:44671: connect: Connection refused (error 111)
I20260925 22:36:30.283706 14565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:30.279556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35430 (local address 127.6.246.252:46735)
0925 22:36:30.279787 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:30.279792 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:30.279806 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:36:30.279860 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:30.279866 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:30.279920 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:30.280082 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:36:30.280564 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:30.281485 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:30.282760 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:30.283170 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:30.283279 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:30.283345 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:30.283514 (+ 169us) server_negotiation.cc:299] Negotiation successful
0925 22:36:30.283591 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":100,"threads_started":1}
I20260925 22:36:30.283710 14563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:30.279108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.252:46735 (local address 127.0.0.1:35430)
0925 22:36:30.279384 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:30.279397 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:30.279523 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:30.280092 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:30.280095 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:30.280112 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:36:30.280236 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:30.280243 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:30.281625 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:30.281628 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:30.282641 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:30.282649 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:30.283279 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:30.283283 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:30.283373 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:30.283415 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 22:36:30.283499 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 22:36:30.283578 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":6,"thread_start_us":85,"threads_started":1}
I20260925 22:36:30.283943 14567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:30.279867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.253:40799 (local address 127.0.0.1:58028)
0925 22:36:30.280304 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:30.280319 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:30.280394 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:30.280881 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:30.280884 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:30.280903 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:36:30.280984 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:30.280990 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:30.282004 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:30.282007 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:30.282855 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:30.282864 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:30.283506 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:30.283509 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:30.283602 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:30.283625 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 22:36:30.283712 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 22:36:30.283810 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":60,"threads_started":1}
I20260925 22:36:30.284268 14048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d481871900894bbdbed9d74795df7ece" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3bf1e94e512547acba5b020a5eab47f5" is_pre_election: true
I20260925 22:36:30.284340 14569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:30.280337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58028 (local address 127.6.246.253:40799)
0925 22:36:30.280640 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:30.280651 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:30.280663 (+ 12us) server_negotiation.cc:407] Connection header received
0925 22:36:30.280703 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:30.280705 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:30.280762 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:30.280871 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:36:30.281099 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:30.281853 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:30.282970 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:30.283392 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:30.283506 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:30.283562 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:30.283724 (+ 162us) server_negotiation.cc:299] Negotiation successful
0925 22:36:30.284256 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":214,"thread_start_us":116,"threads_started":1}
I20260925 22:36:30.284520 14048 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3bf1e94e512547acba5b020a5eab47f5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d481871900894bbdbed9d74795df7ece in term 1.
I20260925 22:36:30.284870 13840 leader_election.cc:302] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3bf1e94e512547acba5b020a5eab47f5, d481871900894bbdbed9d74795df7ece; no voters: f241ea7630914735a806c4b2935b467d
I20260925 22:36:30.284889 13866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3bf1e94e512547acba5b020a5eab47f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d481871900894bbdbed9d74795df7ece" is_pre_election: true
I20260925 22:36:30.285082 13866 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3bf1e94e512547acba5b020a5eab47f5 in term 1.
I20260925 22:36:30.285158 14561 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 22:36:30.285211 14561 raft_consensus.cc:482] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece [term 1 FOLLOWER]: Starting leader election (detected failure of leader f241ea7630914735a806c4b2935b467d)
I20260925 22:36:30.285264 14561 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece [term 1 FOLLOWER]: Advancing to term 2
I20260925 22:36:30.285452 14025 leader_election.cc:302] T 00000000000000000000000000000000 P 3bf1e94e512547acba5b020a5eab47f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3bf1e94e512547acba5b020a5eab47f5, d481871900894bbdbed9d74795df7ece; no voters: f241ea7630914735a806c4b2935b467d
I20260925 22:36:30.285661 14562 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3bf1e94e512547acba5b020a5eab47f5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 22:36:30.285718 14562 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3bf1e94e512547acba5b020a5eab47f5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f241ea7630914735a806c4b2935b467d)
I20260925 22:36:30.285758 14562 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3bf1e94e512547acba5b020a5eab47f5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 22:36:30.286707 14561 raft_consensus.cc:504] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f241ea7630914735a806c4b2935b467d" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 44671 } } peers { permanent_uuid: "d481871900894bbdbed9d74795df7ece" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 40799 } attrs { promote: false } } peers { permanent_uuid: "3bf1e94e512547acba5b020a5eab47f5" member_type: VOTER last_known_addr { host: "127.6.246.252" port: 46735 } attrs { promote: false } }
I20260925 22:36:30.287022 14561 leader_election.cc:288] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece [CANDIDATE]: Term 2 election: Requested vote from peers f241ea7630914735a806c4b2935b467d (127.6.246.254:44671), 3bf1e94e512547acba5b020a5eab47f5 (127.6.246.252:46735)
I20260925 22:36:30.287068 14562 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3bf1e94e512547acba5b020a5eab47f5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f241ea7630914735a806c4b2935b467d" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 44671 } } peers { permanent_uuid: "d481871900894bbdbed9d74795df7ece" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 40799 } attrs { promote: false } } peers { permanent_uuid: "3bf1e94e512547acba5b020a5eab47f5" member_type: VOTER last_known_addr { host: "127.6.246.252" port: 46735 } attrs { promote: false } }
I20260925 22:36:30.287398 14563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:30.287165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:44671 (local address 127.0.0.1:53608)
0925 22:36:30.287277 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:30.287344 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:44671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 22:36:30.287510 14048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d481871900894bbdbed9d74795df7ece" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3bf1e94e512547acba5b020a5eab47f5"
I20260925 22:36:30.287696 14048 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 3bf1e94e512547acba5b020a5eab47f5 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate d481871900894bbdbed9d74795df7ece in current term 2: Already voted for candidate 3bf1e94e512547acba5b020a5eab47f5 in this term.
I20260925 22:36:30.287513 14562 leader_election.cc:288] T 00000000000000000000000000000000 P 3bf1e94e512547acba5b020a5eab47f5 [CANDIDATE]: Term 2 election: Requested vote from peers f241ea7630914735a806c4b2935b467d (127.6.246.254:44671), d481871900894bbdbed9d74795df7ece (127.6.246.253:40799)
I20260925 22:36:30.287772 13866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3bf1e94e512547acba5b020a5eab47f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d481871900894bbdbed9d74795df7ece"
I20260925 22:36:30.287918 13866 raft_consensus.cc:2379] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 3bf1e94e512547acba5b020a5eab47f5 in current term 2: Already voted for candidate d481871900894bbdbed9d74795df7ece in this term.
I20260925 22:36:30.288162 14567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:30.287967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:44671 (local address 127.0.0.1:53630)
0925 22:36:30.288064 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:30.288115 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:44671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 22:36:30.288470 14563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:30.287830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:44671 (local address 127.0.0.1:53618)
0925 22:36:30.288349 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:30.288397 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:44671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455}
W20260925 22:36:30.288605 13840 leader_election.cc:334] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f241ea7630914735a806c4b2935b467d (127.6.246.254:44671): Network error: Client connection negotiation failed: client connection to 127.6.246.254:44671: connect: Connection refused (error 111)
I20260925 22:36:30.288673 13840 leader_election.cc:302] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d481871900894bbdbed9d74795df7ece; no voters: 3bf1e94e512547acba5b020a5eab47f5, f241ea7630914735a806c4b2935b467d
I20260925 22:36:30.288785 14567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:30.288590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:44671 (local address 127.0.0.1:53636)
0925 22:36:30.288670 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:30.288724 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:44671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260925 22:36:30.288798 14561 raft_consensus.cc:2726] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260925 22:36:30.288934 14022 leader_election.cc:334] T 00000000000000000000000000000000 P 3bf1e94e512547acba5b020a5eab47f5 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f241ea7630914735a806c4b2935b467d (127.6.246.254:44671): Network error: Client connection negotiation failed: client connection to 127.6.246.254:44671: connect: Connection refused (error 111)
I20260925 22:36:30.289002 14022 leader_election.cc:302] T 00000000000000000000000000000000 P 3bf1e94e512547acba5b020a5eab47f5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3bf1e94e512547acba5b020a5eab47f5; no voters: d481871900894bbdbed9d74795df7ece, f241ea7630914735a806c4b2935b467d
I20260925 22:36:30.289126 14562 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 3bf1e94e512547acba5b020a5eab47f5 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 22:36:30.769367 14571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:30.768871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:44671 (local address 127.6.246.194:59423)
0925 22:36:30.769161 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:30.769231 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:44671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":102,"threads_started":1}
I20260925 22:36:30.773072 14572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:30.772608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:44671 (local address 127.6.246.195:37285)
0925 22:36:30.772893 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:30.772965 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:44671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":80,"threads_started":1}
I20260925 22:36:30.773514 14573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:30.773012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:44671 (local address 127.6.246.193:57083)
0925 22:36:30.773344 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:30.773407 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:44671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":78,"threads_started":1}
I20260925 22:36:31.770648 14574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:31.770126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:44671 (local address 127.6.246.194:46661)
0925 22:36:31.770430 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:31.770517 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:44671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":78,"threads_started":1}
I20260925 22:36:31.774261 14575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:31.773751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:44671 (local address 127.6.246.195:33763)
0925 22:36:31.774056 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:31.774142 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:44671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":73,"threads_started":1}
I20260925 22:36:31.774546 14576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:31.774127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:44671 (local address 127.6.246.193:60527)
0925 22:36:31.774387 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:31.774455 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:44671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":77,"threads_started":1}
I20260925 22:36:31.893460 14577 raft_consensus.cc:482] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:36:31.893597 14577 raft_consensus.cc:504] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f241ea7630914735a806c4b2935b467d" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 44671 } } peers { permanent_uuid: "d481871900894bbdbed9d74795df7ece" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 40799 } attrs { promote: false } } peers { permanent_uuid: "3bf1e94e512547acba5b020a5eab47f5" member_type: VOTER last_known_addr { host: "127.6.246.252" port: 46735 } attrs { promote: false } }
I20260925 22:36:31.894099 14577 leader_election.cc:288] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f241ea7630914735a806c4b2935b467d (127.6.246.254:44671), 3bf1e94e512547acba5b020a5eab47f5 (127.6.246.252:46735)
I20260925 22:36:31.894687 14578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:31.894303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:44671 (local address 127.0.0.1:53840)
0925 22:36:31.894523 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:31.894602 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:44671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":52,"threads_started":1}
I20260925 22:36:31.894840 14048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d481871900894bbdbed9d74795df7ece" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3bf1e94e512547acba5b020a5eab47f5" is_pre_election: true
I20260925 22:36:31.895012 14048 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3bf1e94e512547acba5b020a5eab47f5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d481871900894bbdbed9d74795df7ece in term 2.
I20260925 22:36:31.895426 13840 leader_election.cc:302] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bf1e94e512547acba5b020a5eab47f5, d481871900894bbdbed9d74795df7ece; no voters:
I20260925 22:36:31.895500 14578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:31.895275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:44671 (local address 127.0.0.1:53854)
0925 22:36:31.895387 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:31.895436 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:44671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260925 22:36:31.895577 13840 leader_election.cc:334] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f241ea7630914735a806c4b2935b467d (127.6.246.254:44671): Network error: Client connection negotiation failed: client connection to 127.6.246.254:44671: connect: Connection refused (error 111)
I20260925 22:36:31.895640 14577 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 22:36:31.895721 14577 raft_consensus.cc:482] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:36:31.895776 14577 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece [term 2 FOLLOWER]: Advancing to term 3
I20260925 22:36:31.896945 14577 raft_consensus.cc:504] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f241ea7630914735a806c4b2935b467d" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 44671 } } peers { permanent_uuid: "d481871900894bbdbed9d74795df7ece" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 40799 } attrs { promote: false } } peers { permanent_uuid: "3bf1e94e512547acba5b020a5eab47f5" member_type: VOTER last_known_addr { host: "127.6.246.252" port: 46735 } attrs { promote: false } }
I20260925 22:36:31.897231 14577 leader_election.cc:288] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece [CANDIDATE]: Term 3 election: Requested vote from peers f241ea7630914735a806c4b2935b467d (127.6.246.254:44671), 3bf1e94e512547acba5b020a5eab47f5 (127.6.246.252:46735)
I20260925 22:36:31.897576 14578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:31.897359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:44671 (local address 127.0.0.1:53856)
0925 22:36:31.897449 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:31.897503 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:44671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 22:36:31.897817 14048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d481871900894bbdbed9d74795df7ece" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3bf1e94e512547acba5b020a5eab47f5"
I20260925 22:36:31.897949 14048 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3bf1e94e512547acba5b020a5eab47f5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 22:36:31.898205 14578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:31.898036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:44671 (local address 127.0.0.1:53858)
0925 22:36:31.898116 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:31.898163 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:44671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260925 22:36:31.898298 13840 leader_election.cc:334] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer f241ea7630914735a806c4b2935b467d (127.6.246.254:44671): Network error: Client connection negotiation failed: client connection to 127.6.246.254:44671: connect: Connection refused (error 111)
I20260925 22:36:31.898986 14048 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3bf1e94e512547acba5b020a5eab47f5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d481871900894bbdbed9d74795df7ece in term 3.
I20260925 22:36:31.899250 13840 leader_election.cc:302] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3bf1e94e512547acba5b020a5eab47f5, d481871900894bbdbed9d74795df7ece; no voters: f241ea7630914735a806c4b2935b467d
I20260925 22:36:31.899405 14577 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece [term 3 FOLLOWER]: Leader election won for term 3
I20260925 22:36:31.899539 14577 raft_consensus.cc:686] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece [term 3 LEADER]: Becoming Leader. State: Replica: d481871900894bbdbed9d74795df7ece, State: Running, Role: LEADER
I20260925 22:36:31.899804 14577 consensus_queue.cc:237] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f241ea7630914735a806c4b2935b467d" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 44671 } } peers { permanent_uuid: "d481871900894bbdbed9d74795df7ece" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 40799 } attrs { promote: false } } peers { permanent_uuid: "3bf1e94e512547acba5b020a5eab47f5" member_type: VOTER last_known_addr { host: "127.6.246.252" port: 46735 } attrs { promote: false } }
I20260925 22:36:31.901084 14580 sys_catalog.cc:455] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece [sys.catalog]: SysCatalogTable state changed. Reason: New leader d481871900894bbdbed9d74795df7ece. Latest consensus state: current_term: 3 leader_uuid: "d481871900894bbdbed9d74795df7ece" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f241ea7630914735a806c4b2935b467d" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 44671 } } peers { permanent_uuid: "d481871900894bbdbed9d74795df7ece" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 40799 } attrs { promote: false } } peers { permanent_uuid: "3bf1e94e512547acba5b020a5eab47f5" member_type: VOTER last_known_addr { host: "127.6.246.252" port: 46735 } attrs { promote: false } } }
I20260925 22:36:31.901171 14580 sys_catalog.cc:458] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece [sys.catalog]: This master's current role is: LEADER
I20260925 22:36:31.901450 14582 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:36:31.903648 14582 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:36:31.904033 14582 catalog_manager.cc:1259] Loaded cluster ID: 2f47f719071c41b3ad8263c885d14031
I20260925 22:36:31.904098 14582 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:36:31.904696 14582 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:36:31.905212 14582 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece: Loaded TSK: 0
I20260925 22:36:31.905892 14582 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:36:31.958704 13856 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58020:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:36:31.961805 13856 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:36:31.964771 14578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:31.964536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:44671 (local address 127.0.0.1:53868)
0925 22:36:31.964649 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:31.964703 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:44671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 22:36:31.964803 14048 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3bf1e94e512547acba5b020a5eab47f5 [term 3 FOLLOWER]: Refusing update from remote peer d481871900894bbdbed9d74795df7ece: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260925 22:36:31.965423 14578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:31.965197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:44671 (local address 127.0.0.1:53872)
0925 22:36:31.965296 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:31.965346 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:44671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 22:36:31.965456 14577 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bf1e94e512547acba5b020a5eab47f5" member_type: VOTER last_known_addr { host: "127.6.246.252" port: 46735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260925 22:36:31.965626 13840 consensus_peers.cc:487] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece -> Peer f241ea7630914735a806c4b2935b467d (127.6.246.254:44671): Couldn't send request to peer f241ea7630914735a806c4b2935b467d. Status: Network error: Client connection negotiation failed: client connection to 127.6.246.254:44671: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 22:36:31.966890 14584 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bf1e94e512547acba5b020a5eab47f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d481871900894bbdbed9d74795df7ece. Latest consensus state: current_term: 3 leader_uuid: "d481871900894bbdbed9d74795df7ece" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f241ea7630914735a806c4b2935b467d" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 44671 } } peers { permanent_uuid: "d481871900894bbdbed9d74795df7ece" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 40799 } attrs { promote: false } } peers { permanent_uuid: "3bf1e94e512547acba5b020a5eab47f5" member_type: VOTER last_known_addr { host: "127.6.246.252" port: 46735 } attrs { promote: false } } }
I20260925 22:36:31.967058 14584 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bf1e94e512547acba5b020a5eab47f5 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:36:31.968192 14577 sys_catalog.cc:455] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "d481871900894bbdbed9d74795df7ece" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f241ea7630914735a806c4b2935b467d" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 44671 } } peers { permanent_uuid: "d481871900894bbdbed9d74795df7ece" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 40799 } attrs { promote: false } } peers { permanent_uuid: "3bf1e94e512547acba5b020a5eab47f5" member_type: VOTER last_known_addr { host: "127.6.246.252" port: 46735 } attrs { promote: false } } }
I20260925 22:36:31.968308 14577 sys_catalog.cc:458] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece [sys.catalog]: This master's current role is: LEADER
I20260925 22:36:31.968513 14585 mvcc.cc:205] Tried to move back new op lower bound from 7333379243883159552 to 7333379243623817216. Current Snapshot: MvccSnapshot[applied={T|T < 7333379243883159552}]
I20260925 22:36:31.968674 14584 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bf1e94e512547acba5b020a5eab47f5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "d481871900894bbdbed9d74795df7ece" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f241ea7630914735a806c4b2935b467d" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 44671 } } peers { permanent_uuid: "d481871900894bbdbed9d74795df7ece" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 40799 } attrs { promote: false } } peers { permanent_uuid: "3bf1e94e512547acba5b020a5eab47f5" member_type: VOTER last_known_addr { host: "127.6.246.252" port: 46735 } attrs { promote: false } } }
I20260925 22:36:31.968801 14584 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bf1e94e512547acba5b020a5eab47f5 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:36:31.978101 14578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:31.973811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.193:42341 (local address 127.0.0.1:33788)
0925 22:36:31.974140 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:31.974155 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:31.974259 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:31.974626 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:31.974629 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:31.974646 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:36:31.974783 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:31.974789 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:31.975902 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:31.975905 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:31.976839 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:31.976848 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:31.977726 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:31.977729 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:31.977827 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:31.977853 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 22:36:31.977902 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 22:36:31.977973 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258}
I20260925 22:36:31.978998 14590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:31.973936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33788 (local address 127.6.246.193:42341)
0925 22:36:31.974141 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:31.974147 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:31.974357 (+ 210us) server_negotiation.cc:407] Connection header received
0925 22:36:31.974440 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:31.974443 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:31.974492 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:31.974885 (+ 393us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:36:31.975048 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:31.975796 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:31.976977 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:31.977590 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:31.978466 (+ 876us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:31.978629 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:31.978700 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 22:36:31.978807 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":76,"threads_started":1}
I20260925 22:36:31.979058 14591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:31.974073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.194:36135 (local address 127.0.0.1:39674)
0925 22:36:31.974281 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:31.974296 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:31.974368 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:31.975420 (+ 1052us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:31.975423 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:31.975442 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:36:31.975527 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:31.975532 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:31.976516 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:31.976519 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:31.977561 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:31.977573 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:31.978482 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:31.978488 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:31.978655 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:31.978683 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 22:36:31.978745 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 22:36:31.978870 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"mutex_wait_us":33,"thread_start_us":99,"threads_started":1}
I20260925 22:36:31.979357 14593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:31.974415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.195:44879 (local address 127.0.0.1:46772)
0925 22:36:31.974638 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:31.974651 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:31.974715 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:31.975291 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:31.975294 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:31.975312 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:36:31.975418 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:31.975424 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:31.976512 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:31.976515 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:31.977717 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:31.977724 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:31.978868 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:31.978872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:31.978959 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:31.978982 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 22:36:31.979056 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 22:36:31.979175 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":74,"thread_start_us":86,"threads_started":1}
I20260925 22:36:31.979589 14592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:31.974407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39674 (local address 127.6.246.194:36135)
0925 22:36:31.975197 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:31.975202 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:31.975215 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:36:31.975266 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:31.975272 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:31.975316 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:31.975410 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:36:31.975639 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:31.976388 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:31.977696 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:31.978319 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:31.979251 (+ 932us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:31.979336 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:31.979391 (+ 55us) server_negotiation.cc:299] Negotiation successful
0925 22:36:31.979482 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":709,"thread_start_us":65,"threads_started":1}
I20260925 22:36:31.979743 14594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:31.974736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46772 (local address 127.6.246.195:44879)
0925 22:36:31.974971 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:31.974977 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:31.974990 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:36:31.975040 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:31.975045 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:31.975095 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:31.975178 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:36:31.975539 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:31.976356 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:31.977833 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:31.978574 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:31.978747 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:31.978809 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:31.979525 (+ 716us) server_negotiation.cc:299] Negotiation successful
0925 22:36:31.979601 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":91,"threads_started":1}
I20260925 22:36:31.979972 14151 tablet_service.cc:1467] Processing CreateTablet for tablet 62cb464f0e9a4430978c63b968e0a10b (DEFAULT_TABLE table=test-table [id=06150e83602548c994c5584e83decf00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:36:31.980137 14288 tablet_service.cc:1467] Processing CreateTablet for tablet 62cb464f0e9a4430978c63b968e0a10b (DEFAULT_TABLE table=test-table [id=06150e83602548c994c5584e83decf00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:36:31.980302 14151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62cb464f0e9a4430978c63b968e0a10b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:36:31.980392 14288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62cb464f0e9a4430978c63b968e0a10b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:36:31.982476 14595 tablet_bootstrap.cc:492] T 62cb464f0e9a4430978c63b968e0a10b P 67b78fd0bb194aa0b4b9cefa890e343f: Bootstrap starting.
I20260925 22:36:31.982666 14596 tablet_bootstrap.cc:492] T 62cb464f0e9a4430978c63b968e0a10b P 8d3bf93a4c6844cf9013e229796c55d1: Bootstrap starting.
I20260925 22:36:31.982846 14426 tablet_service.cc:1467] Processing CreateTablet for tablet 62cb464f0e9a4430978c63b968e0a10b (DEFAULT_TABLE table=test-table [id=06150e83602548c994c5584e83decf00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:36:31.983108 14426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62cb464f0e9a4430978c63b968e0a10b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:36:31.983424 14595 tablet_bootstrap.cc:654] T 62cb464f0e9a4430978c63b968e0a10b P 67b78fd0bb194aa0b4b9cefa890e343f: Neither blocks nor log segments found. Creating new log.
I20260925 22:36:31.983565 14596 tablet_bootstrap.cc:654] T 62cb464f0e9a4430978c63b968e0a10b P 8d3bf93a4c6844cf9013e229796c55d1: Neither blocks nor log segments found. Creating new log.
I20260925 22:36:31.984694 14595 tablet_bootstrap.cc:492] T 62cb464f0e9a4430978c63b968e0a10b P 67b78fd0bb194aa0b4b9cefa890e343f: No bootstrap required, opened a new log
I20260925 22:36:31.984695 14596 tablet_bootstrap.cc:492] T 62cb464f0e9a4430978c63b968e0a10b P 8d3bf93a4c6844cf9013e229796c55d1: No bootstrap required, opened a new log
I20260925 22:36:31.984777 14596 ts_tablet_manager.cc:1397] T 62cb464f0e9a4430978c63b968e0a10b P 8d3bf93a4c6844cf9013e229796c55d1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 22:36:31.984777 14595 ts_tablet_manager.cc:1397] T 62cb464f0e9a4430978c63b968e0a10b P 67b78fd0bb194aa0b4b9cefa890e343f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260925 22:36:31.985174 14596 raft_consensus.cc:348] T 62cb464f0e9a4430978c63b968e0a10b P 8d3bf93a4c6844cf9013e229796c55d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b78fd0bb194aa0b4b9cefa890e343f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 42341 } } peers { permanent_uuid: "8d3bf93a4c6844cf9013e229796c55d1" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 36135 } } peers { permanent_uuid: "d941f16284f0429381d37c34d0eaae2a" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 44879 } }
I20260925 22:36:31.985179 14595 raft_consensus.cc:348] T 62cb464f0e9a4430978c63b968e0a10b P 67b78fd0bb194aa0b4b9cefa890e343f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b78fd0bb194aa0b4b9cefa890e343f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 42341 } } peers { permanent_uuid: "8d3bf93a4c6844cf9013e229796c55d1" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 36135 } } peers { permanent_uuid: "d941f16284f0429381d37c34d0eaae2a" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 44879 } }
I20260925 22:36:31.985280 14596 raft_consensus.cc:374] T 62cb464f0e9a4430978c63b968e0a10b P 8d3bf93a4c6844cf9013e229796c55d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:36:31.985280 14595 raft_consensus.cc:374] T 62cb464f0e9a4430978c63b968e0a10b P 67b78fd0bb194aa0b4b9cefa890e343f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:36:31.985312 14595 raft_consensus.cc:729] T 62cb464f0e9a4430978c63b968e0a10b P 67b78fd0bb194aa0b4b9cefa890e343f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67b78fd0bb194aa0b4b9cefa890e343f, State: Initialized, Role: FOLLOWER
I20260925 22:36:31.985312 14596 raft_consensus.cc:729] T 62cb464f0e9a4430978c63b968e0a10b P 8d3bf93a4c6844cf9013e229796c55d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d3bf93a4c6844cf9013e229796c55d1, State: Initialized, Role: FOLLOWER
I20260925 22:36:31.985440 14595 consensus_queue.cc:260] T 62cb464f0e9a4430978c63b968e0a10b P 67b78fd0bb194aa0b4b9cefa890e343f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b78fd0bb194aa0b4b9cefa890e343f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 42341 } } peers { permanent_uuid: "8d3bf93a4c6844cf9013e229796c55d1" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 36135 } } peers { permanent_uuid: "d941f16284f0429381d37c34d0eaae2a" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 44879 } }
I20260925 22:36:31.985440 14596 consensus_queue.cc:260] T 62cb464f0e9a4430978c63b968e0a10b P 8d3bf93a4c6844cf9013e229796c55d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b78fd0bb194aa0b4b9cefa890e343f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 42341 } } peers { permanent_uuid: "8d3bf93a4c6844cf9013e229796c55d1" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 36135 } } peers { permanent_uuid: "d941f16284f0429381d37c34d0eaae2a" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 44879 } }
I20260925 22:36:31.985634 14599 tablet_bootstrap.cc:492] T 62cb464f0e9a4430978c63b968e0a10b P d941f16284f0429381d37c34d0eaae2a: Bootstrap starting.
I20260925 22:36:31.985787 14596 ts_tablet_manager.cc:1428] T 62cb464f0e9a4430978c63b968e0a10b P 8d3bf93a4c6844cf9013e229796c55d1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:36:31.985781 14595 ts_tablet_manager.cc:1428] T 62cb464f0e9a4430978c63b968e0a10b P 67b78fd0bb194aa0b4b9cefa890e343f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260925 22:36:31.986439 14574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:31.986200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:44671 (local address 127.6.246.194:44915)
0925 22:36:31.986303 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:31.986363 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:44671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 22:36:31.987385 14576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:31.986879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:44671 (local address 127.6.246.193:57551)
0925 22:36:31.987240 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:31.987300 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:44671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293}
I20260925 22:36:31.987771 14574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:31.987434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:44671 (local address 127.6.246.194:56621)
0925 22:36:31.987653 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:31.987702 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:44671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158}
I20260925 22:36:31.988524 14599 tablet_bootstrap.cc:654] T 62cb464f0e9a4430978c63b968e0a10b P d941f16284f0429381d37c34d0eaae2a: Neither blocks nor log segments found. Creating new log.
I20260925 22:36:31.989822 14599 tablet_bootstrap.cc:492] T 62cb464f0e9a4430978c63b968e0a10b P d941f16284f0429381d37c34d0eaae2a: No bootstrap required, opened a new log
I20260925 22:36:31.989919 14599 ts_tablet_manager.cc:1397] T 62cb464f0e9a4430978c63b968e0a10b P d941f16284f0429381d37c34d0eaae2a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260925 22:36:31.990242 14599 raft_consensus.cc:348] T 62cb464f0e9a4430978c63b968e0a10b P d941f16284f0429381d37c34d0eaae2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b78fd0bb194aa0b4b9cefa890e343f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 42341 } } peers { permanent_uuid: "8d3bf93a4c6844cf9013e229796c55d1" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 36135 } } peers { permanent_uuid: "d941f16284f0429381d37c34d0eaae2a" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 44879 } }
I20260925 22:36:31.990353 14599 raft_consensus.cc:374] T 62cb464f0e9a4430978c63b968e0a10b P d941f16284f0429381d37c34d0eaae2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:36:31.990396 14599 raft_consensus.cc:729] T 62cb464f0e9a4430978c63b968e0a10b P d941f16284f0429381d37c34d0eaae2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d941f16284f0429381d37c34d0eaae2a, State: Initialized, Role: FOLLOWER
I20260925 22:36:31.990520 14599 consensus_queue.cc:260] T 62cb464f0e9a4430978c63b968e0a10b P d941f16284f0429381d37c34d0eaae2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b78fd0bb194aa0b4b9cefa890e343f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 42341 } } peers { permanent_uuid: "8d3bf93a4c6844cf9013e229796c55d1" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 36135 } } peers { permanent_uuid: "d941f16284f0429381d37c34d0eaae2a" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 44879 } }
I20260925 22:36:31.990810 14599 ts_tablet_manager.cc:1428] T 62cb464f0e9a4430978c63b968e0a10b P d941f16284f0429381d37c34d0eaae2a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:36:31.991526 14575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:31.991117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:44671 (local address 127.6.246.195:53703)
0925 22:36:31.991412 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:31.991461 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:44671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224}
I20260925 22:36:31.992173 14575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:31.991942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:44671 (local address 127.6.246.195:44387)
0925 22:36:31.992065 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:31.992120 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:44671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260925 22:36:31.992238 14494 heartbeater.cc:503] Master 127.6.246.253:40799 was elected leader, sending a full tablet report...
I20260925 22:36:32.047987 14603 raft_consensus.cc:482] T 62cb464f0e9a4430978c63b968e0a10b P d941f16284f0429381d37c34d0eaae2a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:36:32.048167 14603 raft_consensus.cc:504] T 62cb464f0e9a4430978c63b968e0a10b P d941f16284f0429381d37c34d0eaae2a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b78fd0bb194aa0b4b9cefa890e343f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 42341 } } peers { permanent_uuid: "8d3bf93a4c6844cf9013e229796c55d1" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 36135 } } peers { permanent_uuid: "d941f16284f0429381d37c34d0eaae2a" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 44879 } }
I20260925 22:36:32.049885 14603 leader_election.cc:288] T 62cb464f0e9a4430978c63b968e0a10b P d941f16284f0429381d37c34d0eaae2a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67b78fd0bb194aa0b4b9cefa890e343f (127.6.246.193:42341), 8d3bf93a4c6844cf9013e229796c55d1 (127.6.246.194:36135)
I20260925 22:36:32.054947 14590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:32.050398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.195:50711 (local address 127.6.246.193:42341)
0925 22:36:32.050548 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:32.050554 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:32.050572 (+ 18us) server_negotiation.cc:407] Connection header received
0925 22:36:32.050638 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:32.050642 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:32.050702 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:32.050788 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:36:32.051254 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:32.052151 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:32.053856 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:32.054364 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:32.054518 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:32.054577 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:32.054746 (+ 169us) server_negotiation.cc:299] Negotiation successful
0925 22:36:32.054828 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 22:36:32.054947 14575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:32.050097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.193:42341 (local address 127.6.246.195:50711)
0925 22:36:32.050243 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:32.050275 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:32.050387 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:32.050938 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:32.050945 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:32.050970 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:36:32.051104 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:32.051112 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:32.052278 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:32.052283 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:32.053663 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:32.053673 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:32.054496 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:32.054500 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:32.054607 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:32.054631 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 22:36:32.054731 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 22:36:32.054803 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":44}
I20260925 22:36:32.055335 14592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:32.051276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.195:51713 (local address 127.6.246.194:36135)
0925 22:36:32.051444 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:32.051448 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:32.051598 (+ 150us) server_negotiation.cc:407] Connection header received
0925 22:36:32.051870 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:32.051872 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:32.051968 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:32.052069 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:36:32.052447 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:32.053232 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:32.054343 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:32.054742 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:32.054864 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:32.054912 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:32.055150 (+ 238us) server_negotiation.cc:299] Negotiation successful
0925 22:36:32.055222 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260925 22:36:32.055567 14171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62cb464f0e9a4430978c63b968e0a10b" candidate_uuid: "d941f16284f0429381d37c34d0eaae2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b78fd0bb194aa0b4b9cefa890e343f" is_pre_election: true
I20260925 22:36:32.055713 14171 raft_consensus.cc:2454] T 62cb464f0e9a4430978c63b968e0a10b P 67b78fd0bb194aa0b4b9cefa890e343f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d941f16284f0429381d37c34d0eaae2a in term 0.
I20260925 22:36:32.056063 14604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:32.050823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.194:36135 (local address 127.6.246.195:51713)
0925 22:36:32.051504 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:32.051517 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:32.051674 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:32.052176 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:32.052178 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:32.052194 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:36:32.052319 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:32.052326 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:32.053368 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:32.053371 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:32.054223 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:32.054232 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:32.054869 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:32.054873 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:32.054976 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:32.055018 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 22:36:32.055906 (+ 888us) client_negotiation.cc:240] Negotiation successful
0925 22:36:32.055964 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"mutex_wait_us":1,"thread_start_us":130,"threads_started":1}
I20260925 22:36:32.056111 14379 leader_election.cc:302] T 62cb464f0e9a4430978c63b968e0a10b P d941f16284f0429381d37c34d0eaae2a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b78fd0bb194aa0b4b9cefa890e343f, d941f16284f0429381d37c34d0eaae2a; no voters:
I20260925 22:36:32.056476 14603 raft_consensus.cc:2781] T 62cb464f0e9a4430978c63b968e0a10b P d941f16284f0429381d37c34d0eaae2a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:36:32.056541 14603 raft_consensus.cc:482] T 62cb464f0e9a4430978c63b968e0a10b P d941f16284f0429381d37c34d0eaae2a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:36:32.056592 14603 raft_consensus.cc:3037] T 62cb464f0e9a4430978c63b968e0a10b P d941f16284f0429381d37c34d0eaae2a [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:36:32.056648 14308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62cb464f0e9a4430978c63b968e0a10b" candidate_uuid: "d941f16284f0429381d37c34d0eaae2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d3bf93a4c6844cf9013e229796c55d1" is_pre_election: true
I20260925 22:36:32.056911 14308 raft_consensus.cc:2454] T 62cb464f0e9a4430978c63b968e0a10b P 8d3bf93a4c6844cf9013e229796c55d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d941f16284f0429381d37c34d0eaae2a in term 0.
I20260925 22:36:32.057706 14603 raft_consensus.cc:504] T 62cb464f0e9a4430978c63b968e0a10b P d941f16284f0429381d37c34d0eaae2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b78fd0bb194aa0b4b9cefa890e343f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 42341 } } peers { permanent_uuid: "8d3bf93a4c6844cf9013e229796c55d1" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 36135 } } peers { permanent_uuid: "d941f16284f0429381d37c34d0eaae2a" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 44879 } }
I20260925 22:36:32.058034 14603 leader_election.cc:288] T 62cb464f0e9a4430978c63b968e0a10b P d941f16284f0429381d37c34d0eaae2a [CANDIDATE]: Term 1 election: Requested vote from peers 67b78fd0bb194aa0b4b9cefa890e343f (127.6.246.193:42341), 8d3bf93a4c6844cf9013e229796c55d1 (127.6.246.194:36135)
I20260925 22:36:32.058378 14171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62cb464f0e9a4430978c63b968e0a10b" candidate_uuid: "d941f16284f0429381d37c34d0eaae2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b78fd0bb194aa0b4b9cefa890e343f"
I20260925 22:36:32.058519 14171 raft_consensus.cc:3037] T 62cb464f0e9a4430978c63b968e0a10b P 67b78fd0bb194aa0b4b9cefa890e343f [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:36:32.058492 14308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62cb464f0e9a4430978c63b968e0a10b" candidate_uuid: "d941f16284f0429381d37c34d0eaae2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d3bf93a4c6844cf9013e229796c55d1"
I20260925 22:36:32.058627 14308 raft_consensus.cc:3037] T 62cb464f0e9a4430978c63b968e0a10b P 8d3bf93a4c6844cf9013e229796c55d1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:36:32.059623 14171 raft_consensus.cc:2454] T 62cb464f0e9a4430978c63b968e0a10b P 67b78fd0bb194aa0b4b9cefa890e343f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d941f16284f0429381d37c34d0eaae2a in term 1.
I20260925 22:36:32.059715 14308 raft_consensus.cc:2454] T 62cb464f0e9a4430978c63b968e0a10b P 8d3bf93a4c6844cf9013e229796c55d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d941f16284f0429381d37c34d0eaae2a in term 1.
I20260925 22:36:32.059958 14379 leader_election.cc:302] T 62cb464f0e9a4430978c63b968e0a10b P d941f16284f0429381d37c34d0eaae2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b78fd0bb194aa0b4b9cefa890e343f, d941f16284f0429381d37c34d0eaae2a; no voters:
I20260925 22:36:32.060135 14603 raft_consensus.cc:2781] T 62cb464f0e9a4430978c63b968e0a10b P d941f16284f0429381d37c34d0eaae2a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:36:32.060351 14603 raft_consensus.cc:686] T 62cb464f0e9a4430978c63b968e0a10b P d941f16284f0429381d37c34d0eaae2a [term 1 LEADER]: Becoming Leader. State: Replica: d941f16284f0429381d37c34d0eaae2a, State: Running, Role: LEADER
I20260925 22:36:32.060721 14603 consensus_queue.cc:237] T 62cb464f0e9a4430978c63b968e0a10b P d941f16284f0429381d37c34d0eaae2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b78fd0bb194aa0b4b9cefa890e343f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 42341 } } peers { permanent_uuid: "8d3bf93a4c6844cf9013e229796c55d1" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 36135 } } peers { permanent_uuid: "d941f16284f0429381d37c34d0eaae2a" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 44879 } }
I20260925 22:36:32.061024 14604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:32.060800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:44671 (local address 127.6.246.195:60429)
0925 22:36:32.060913 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:32.060965 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:44671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260925 22:36:32.063304 13856 catalog_manager.cc:5495] T 62cb464f0e9a4430978c63b968e0a10b P d941f16284f0429381d37c34d0eaae2a reported cstate change: term changed from 0 to 1, leader changed from <none> to d941f16284f0429381d37c34d0eaae2a (127.6.246.195). New cstate: current_term: 1 leader_uuid: "d941f16284f0429381d37c34d0eaae2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b78fd0bb194aa0b4b9cefa890e343f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 42341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d3bf93a4c6844cf9013e229796c55d1" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 36135 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d941f16284f0429381d37c34d0eaae2a" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 44879 } health_report { overall_health: HEALTHY } } }
I20260925 22:36:32.076463 14594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:32.072833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46776 (local address 127.6.246.195:44879)
0925 22:36:32.072938 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:32.072941 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:32.073005 (+ 64us) server_negotiation.cc:407] Connection header received
0925 22:36:32.073133 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:32.073135 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:32.073184 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:32.073265 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:36:32.073513 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:32.074031 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:32.075191 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:32.075398 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:32.075474 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:32.075722 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:36:32.076116 (+ 394us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:36:32.076196 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:32.076282 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 22:36:32.076324 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 22:36:32.079010 14171 raft_consensus.cc:1261] T 62cb464f0e9a4430978c63b968e0a10b P 67b78fd0bb194aa0b4b9cefa890e343f [term 1 FOLLOWER]: Refusing update from remote peer d941f16284f0429381d37c34d0eaae2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:36:32.079320 14308 raft_consensus.cc:1261] T 62cb464f0e9a4430978c63b968e0a10b P 8d3bf93a4c6844cf9013e229796c55d1 [term 1 FOLLOWER]: Refusing update from remote peer d941f16284f0429381d37c34d0eaae2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:36:32.079504 14603 consensus_queue.cc:1035] T 62cb464f0e9a4430978c63b968e0a10b P d941f16284f0429381d37c34d0eaae2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b78fd0bb194aa0b4b9cefa890e343f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 42341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 22:36:32.079890 14495 tablet.cc:2369] T 62cb464f0e9a4430978c63b968e0a10b P d941f16284f0429381d37c34d0eaae2a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:36:32.079923 14603 consensus_queue.cc:1035] T 62cb464f0e9a4430978c63b968e0a10b P d941f16284f0429381d37c34d0eaae2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d3bf93a4c6844cf9013e229796c55d1" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 36135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:36:32.080667 14219 heartbeater.cc:503] Master 127.6.246.253:40799 was elected leader, sending a full tablet report...
I20260925 22:36:32.081357 14576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:32.081061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:44671 (local address 127.6.246.193:35365)
0925 22:36:32.081233 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:32.081287 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:44671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260925 22:36:32.082121 14356 heartbeater.cc:503] Master 127.6.246.253:40799 was elected leader, sending a full tablet report...
I20260925 22:36:32.083026 14574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:32.082632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:44671 (local address 127.6.246.194:55519)
0925 22:36:32.082901 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:32.082959 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:44671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204}
I20260925 22:36:32.085388 14604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:32.085177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:44671 (local address 127.6.246.195:44859)
0925 22:36:32.085265 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:32.085316 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:44671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260925 22:36:32.085610 14576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:32.085130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:44671 (local address 127.6.246.193:49595)
0925 22:36:32.085481 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:32.085548 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:44671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":283}
I20260925 22:36:32.086238 14604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:32.085996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:44671 (local address 127.6.246.195:54753)
0925 22:36:32.086124 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:32.086174 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:44671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260925 22:36:32.087258 14574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:32.086860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:44671 (local address 127.6.246.194:51679)
0925 22:36:32.087143 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:32.087194 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:44671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201}
I20260925 22:36:32.094139 13856 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58020:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:36:32.094344 13856 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58020:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:36:32.097723 13856 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d481871900894bbdbed9d74795df7ece: Sending DeleteTablet for 3 replicas of tablet 62cb464f0e9a4430978c63b968e0a10b
I20260925 22:36:32.098318 14151 tablet_service.cc:1514] Processing DeleteTablet for tablet 62cb464f0e9a4430978c63b968e0a10b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:36:32 UTC) from {username='oryx', principal='oryx/127.6.246.253@KRBTEST.COM'} at 127.0.0.1:33788
I20260925 22:36:32.098598 14288 tablet_service.cc:1514] Processing DeleteTablet for tablet 62cb464f0e9a4430978c63b968e0a10b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:36:32 UTC) from {username='oryx', principal='oryx/127.6.246.253@KRBTEST.COM'} at 127.0.0.1:39674
I20260925 22:36:32.098616 14426 tablet_service.cc:1514] Processing DeleteTablet for tablet 62cb464f0e9a4430978c63b968e0a10b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:36:32 UTC) from {username='oryx', principal='oryx/127.6.246.253@KRBTEST.COM'} at 127.0.0.1:46772
I20260925 22:36:32.098928 14621 tablet_replica.cc:330] T 62cb464f0e9a4430978c63b968e0a10b P 8d3bf93a4c6844cf9013e229796c55d1: stopping tablet replica
I20260925 22:36:32.099113 14621 raft_consensus.cc:2229] T 62cb464f0e9a4430978c63b968e0a10b P 8d3bf93a4c6844cf9013e229796c55d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:36:32.099267 14620 tablet_replica.cc:330] T 62cb464f0e9a4430978c63b968e0a10b P 67b78fd0bb194aa0b4b9cefa890e343f: stopping tablet replica
I20260925 22:36:32.099334 14621 raft_consensus.cc:2258] T 62cb464f0e9a4430978c63b968e0a10b P 8d3bf93a4c6844cf9013e229796c55d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:36:32.099545 14622 tablet_replica.cc:330] T 62cb464f0e9a4430978c63b968e0a10b P d941f16284f0429381d37c34d0eaae2a: stopping tablet replica
I20260925 22:36:32.099695 14622 raft_consensus.cc:2229] T 62cb464f0e9a4430978c63b968e0a10b P d941f16284f0429381d37c34d0eaae2a [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:36:32.099696 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 14000
I20260925 22:36:32.100028 14622 raft_consensus.cc:2258] T 62cb464f0e9a4430978c63b968e0a10b P d941f16284f0429381d37c34d0eaae2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:36:32.101459 14621 ts_tablet_manager.cc:1905] T 62cb464f0e9a4430978c63b968e0a10b P 8d3bf93a4c6844cf9013e229796c55d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:36:32.102761 14622 ts_tablet_manager.cc:1905] T 62cb464f0e9a4430978c63b968e0a10b P d941f16284f0429381d37c34d0eaae2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:36:32.104701 14621 ts_tablet_manager.cc:1918] T 62cb464f0e9a4430978c63b968e0a10b P 8d3bf93a4c6844cf9013e229796c55d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:36:32.104822 14621 log.cc:1196] T 62cb464f0e9a4430978c63b968e0a10b P 8d3bf93a4c6844cf9013e229796c55d1: Deleting WAL directory at /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-1/wal/wals/62cb464f0e9a4430978c63b968e0a10b
I20260925 22:36:32.105209 14621 ts_tablet_manager.cc:1939] T 62cb464f0e9a4430978c63b968e0a10b P 8d3bf93a4c6844cf9013e229796c55d1: Deleting consensus metadata
I20260925 22:36:32.105453 14622 ts_tablet_manager.cc:1918] T 62cb464f0e9a4430978c63b968e0a10b P d941f16284f0429381d37c34d0eaae2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:36:32.105618 14622 log.cc:1196] T 62cb464f0e9a4430978c63b968e0a10b P d941f16284f0429381d37c34d0eaae2a: Deleting WAL directory at /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790375752797504-7131-0/minicluster-data/ts-2/wal/wals/62cb464f0e9a4430978c63b968e0a10b
I20260925 22:36:32.105970 14622 ts_tablet_manager.cc:1939] T 62cb464f0e9a4430978c63b968e0a10b P d941f16284f0429381d37c34d0eaae2a: Deleting consensus metadata
I20260925 22:36:32.106215 13840 catalog_manager.cc:4841] TS 8d3bf93a4c6844cf9013e229796c55d1 (127.6.246.194:36135): tablet 62cb464f0e9a4430978c63b968e0a10b (table test-table [id=06150e83602548c994c5584e83decf00]) successfully deleted
I20260925 22:36:32.106590 13843 catalog_manager.cc:4841] TS d941f16284f0429381d37c34d0eaae2a (127.6.246.195:44879): tablet 62cb464f0e9a4430978c63b968e0a10b (table test-table [id=06150e83602548c994c5584e83decf00]) successfully deleted
I20260925 22:36:32.109627 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 14225
I20260925 22:36:32.109839 14593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:32.109572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.193:42341 (local address 127.0.0.1:33790)
0925 22:36:32.109704 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:32.109756 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.193:42341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 22:36:32.110721 13842 catalog_manager.cc:4584] TS 67b78fd0bb194aa0b4b9cefa890e343f (127.6.246.193:42341): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 62cb464f0e9a4430978c63b968e0a10b: Network error: Client connection negotiation failed: client connection to 127.6.246.193:42341: connect: Connection refused (error 111)
I20260925 22:36:32.117632 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 14361
I20260925 22:36:32.123575 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 13719
I20260925 22:36:32.130119 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 13828
2026-09-25T22:36:32Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6863 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:36:32 dist-test-slave-wjdr krb5kdc[14625](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:36:32 dist-test-slave-wjdr krb5kdc[14625](info): set up 2 sockets
Sep 25 22:36:32 dist-test-slave-wjdr krb5kdc[14625](info): commencing operation
krb5kdc: starting...
W20260925 22:36:33.169091 7131 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Sep 25 22:36:33 dist-test-slave-wjdr krb5kdc[14625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375793, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:36:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:36:33Z Disabled control of system clock
WARNING: no policy specified for oryx/127.6.246.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.246.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.246.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/krb5kdc/oryx_127.6.246.254.keytab.
Entry for principal oryx/127.6.246.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/krb5kdc/oryx_127.6.246.254.keytab.
I20260925 22:36:33.283277 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:38821
--webserver_interface=127.6.246.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/krb5kdc/oryx_127.6.246.254.keytab
--principal=oryx/127.6.246.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:35249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.246.254:38821
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:36:33.389627 14638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:36:33.389873 14638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:36:33.389914 14638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:36:33.393786 14638 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:36:33.393837 14638 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 22:36:33.393854 14638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:36:33.393872 14638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:36:33.393888 14638 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:36:33.393903 14638 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:36:33.398904 14638 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:35249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.246.254:38821
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/krb5kdc/oryx_127.6.246.254.keytab
--principal=oryx/127.6.246.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:38821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.246.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:36:33.399794 14638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:36:33.400722 14638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:36:33.408870 14646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:33.408977 14638 server_base.cc:1034] running on GCE node
W20260925 22:36:33.408880 14643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:33.408880 14644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:33.409689 14638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:36:33.410744 14638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:36:33.411893 14638 hybrid_clock.cc:648] HybridClock initialized: now 1790375793411869 us; error 37 us; skew 500 ppm
Sep 25 22:36:33 dist-test-slave-wjdr krb5kdc[14625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375793, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.246.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:36:33.414603 14638 init.cc:377] Logged in from keytab as oryx/127.6.246.254@KRBTEST.COM (short username oryx)
I20260925 22:36:33.415488 14638 webserver.cc:460] Webserver started at http://127.6.246.254:33511/ using document root <none> and password file <none>
I20260925 22:36:33.416000 14638 fs_manager.cc:359] Metadata directory not provided
I20260925 22:36:33.416059 14638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:36:33.416251 14638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:36:33.418205 14638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/minicluster-data/master-0/data/instance:
uuid: "302768cab249462291567a64833dd1a1"
format_stamp: "Formatted at 2026-09-25 22:36:33 on dist-test-slave-wjdr"
server_key: "b7e36b97617083eff01448ab6889f5f2"
server_key_iv: "545be085333abc84f2b84abc496c36e2"
server_key_version: "encryptionkey@0"
I20260925 22:36:33.418752 14638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/minicluster-data/master-0/wal/instance:
uuid: "302768cab249462291567a64833dd1a1"
format_stamp: "Formatted at 2026-09-25 22:36:33 on dist-test-slave-wjdr"
server_key: "b7e36b97617083eff01448ab6889f5f2"
server_key_iv: "545be085333abc84f2b84abc496c36e2"
server_key_version: "encryptionkey@0"
I20260925 22:36:33.422190 14638 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 22:36:33.424413 14653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:33.425595 14638 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:36:33.425738 14638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/minicluster-data/master-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/minicluster-data/master-0/wal
uuid: "302768cab249462291567a64833dd1a1"
format_stamp: "Formatted at 2026-09-25 22:36:33 on dist-test-slave-wjdr"
server_key: "b7e36b97617083eff01448ab6889f5f2"
server_key_iv: "545be085333abc84f2b84abc496c36e2"
server_key_version: "encryptionkey@0"
I20260925 22:36:33.425856 14638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:36:33.445698 14638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:36:33.448376 14638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:36:33.448762 14638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:36:33.456621 14638 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.254:38821
I20260925 22:36:33.456646 14715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.254:38821 every 8 connection(s)
I20260925 22:36:33.457474 14638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
I20260925 22:36:33.458941 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 14638
I20260925 22:36:33.459085 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/minicluster-data/master-0/wal/instance
I20260925 22:36:33.459403 7131 external_mini_cluster.cc:1428] Setting key 9dc941bd4b5aa9c5da3e628142a3dfd8
I20260925 22:36:33.460356 14716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:36:33.465553 14716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 302768cab249462291567a64833dd1a1: Bootstrap starting.
Sep 25 22:36:33 dist-test-slave-wjdr krb5kdc[14625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375793, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM
I20260925 22:36:33.467945 14716 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 302768cab249462291567a64833dd1a1: Neither blocks nor log segments found. Creating new log.
I20260925 22:36:33.468791 14716 log.cc:824] T 00000000000000000000000000000000 P 302768cab249462291567a64833dd1a1: Log is configured to *not* fsync() on all Append() calls
I20260925 22:36:33.470805 14716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 302768cab249462291567a64833dd1a1: No bootstrap required, opened a new log
I20260925 22:36:33.473600 14719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:33.460724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50306 (local address 127.6.246.254:38821)
0925 22:36:33.461130 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:33.461144 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:33.461175 (+ 31us) server_negotiation.cc:407] Connection header received
0925 22:36:33.461774 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:33.461788 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:33.462087 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:33.462401 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:33.463162 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:33.464121 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:33.465003 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:33.465464 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:33.465629 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:33.468230 (+ 2601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:33.468235 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:33.468246 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:33.468271 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:33.470948 (+ 2677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:33.471404 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:33.471408 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:33.471412 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:33.471511 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:33.471810 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:33.471813 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:33.471814 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:33.472176 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:33.472320 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:33.472627 (+ 307us) server_negotiation.cc:299] Negotiation successful
0925 22:36:33.472941 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":139,"threads_started":1}
I20260925 22:36:33.474146 14716 raft_consensus.cc:348] T 00000000000000000000000000000000 P 302768cab249462291567a64833dd1a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "302768cab249462291567a64833dd1a1" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 38821 } }
I20260925 22:36:33.474344 14716 raft_consensus.cc:374] T 00000000000000000000000000000000 P 302768cab249462291567a64833dd1a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:36:33.474447 14716 raft_consensus.cc:729] T 00000000000000000000000000000000 P 302768cab249462291567a64833dd1a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 302768cab249462291567a64833dd1a1, State: Initialized, Role: FOLLOWER
I20260925 22:36:33.474901 14716 consensus_queue.cc:260] T 00000000000000000000000000000000 P 302768cab249462291567a64833dd1a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "302768cab249462291567a64833dd1a1" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 38821 } }
I20260925 22:36:33.475046 14716 raft_consensus.cc:388] T 00000000000000000000000000000000 P 302768cab249462291567a64833dd1a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:36:33.475108 14716 raft_consensus.cc:482] T 00000000000000000000000000000000 P 302768cab249462291567a64833dd1a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:36:33.475195 14716 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 302768cab249462291567a64833dd1a1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:36:33.476030 14716 raft_consensus.cc:504] T 00000000000000000000000000000000 P 302768cab249462291567a64833dd1a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "302768cab249462291567a64833dd1a1" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 38821 } }
I20260925 22:36:33.476294 14716 leader_election.cc:302] T 00000000000000000000000000000000 P 302768cab249462291567a64833dd1a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 302768cab249462291567a64833dd1a1; no voters:
I20260925 22:36:33.476569 14716 leader_election.cc:288] T 00000000000000000000000000000000 P 302768cab249462291567a64833dd1a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:36:33.476696 14721 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 302768cab249462291567a64833dd1a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:36:33.476929 14721 raft_consensus.cc:686] T 00000000000000000000000000000000 P 302768cab249462291567a64833dd1a1 [term 1 LEADER]: Becoming Leader. State: Replica: 302768cab249462291567a64833dd1a1, State: Running, Role: LEADER
I20260925 22:36:33.477272 14721 consensus_queue.cc:237] T 00000000000000000000000000000000 P 302768cab249462291567a64833dd1a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "302768cab249462291567a64833dd1a1" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 38821 } }
I20260925 22:36:33.477699 14716 sys_catalog.cc:564] T 00000000000000000000000000000000 P 302768cab249462291567a64833dd1a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:36:33.478933 14722 sys_catalog.cc:455] T 00000000000000000000000000000000 P 302768cab249462291567a64833dd1a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "302768cab249462291567a64833dd1a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "302768cab249462291567a64833dd1a1" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 38821 } } }
I20260925 22:36:33.478935 14723 sys_catalog.cc:455] T 00000000000000000000000000000000 P 302768cab249462291567a64833dd1a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 302768cab249462291567a64833dd1a1. Latest consensus state: current_term: 1 leader_uuid: "302768cab249462291567a64833dd1a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "302768cab249462291567a64833dd1a1" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 38821 } } }
I20260925 22:36:33.479103 14722 sys_catalog.cc:458] T 00000000000000000000000000000000 P 302768cab249462291567a64833dd1a1 [sys.catalog]: This master's current role is: LEADER
I20260925 22:36:33.479137 14723 sys_catalog.cc:458] T 00000000000000000000000000000000 P 302768cab249462291567a64833dd1a1 [sys.catalog]: This master's current role is: LEADER
I20260925 22:36:33.479501 14732 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:36:33.482012 14732 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:36:33.487063 14732 catalog_manager.cc:1347] Generated new cluster ID: cac7fa13a5ab452381b70b6de6d2b4f4
I20260925 22:36:33.487123 14732 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:36:33.515048 14732 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:36:33.516242 14732 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:36:33.522580 14732 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 302768cab249462291567a64833dd1a1: Generated new TSK 0
I20260925 22:36:33.523182 14732 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.6.246.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.246.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.246.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/krb5kdc/oryx_127.6.246.193.keytab.
Entry for principal oryx/127.6.246.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/krb5kdc/oryx_127.6.246.193.keytab.
I20260925 22:36:33.550923 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.193:0
--local_ip_for_outbound_sockets=127.6.246.193
--webserver_interface=127.6.246.193
--webserver_port=0
--tserver_master_addrs=127.6.246.254:38821
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/krb5kdc/oryx_127.6.246.193.keytab
--principal=oryx/127.6.246.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:35249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:36:33.659260 14742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:36:33.659543 14742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:36:33.659617 14742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:36:33.663559 14742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:36:33.663640 14742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:36:33.663705 14742 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 22:36:33.663779 14742 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.193
I20260925 22:36:33.668954 14742 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:35249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/krb5kdc/oryx_127.6.246.193.keytab
--principal=oryx/127.6.246.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.246.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.246.254:38821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14742
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.193
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:36:33.669968 14742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:36:33.670964 14742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:36:33.680617 14747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:33.680712 14748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:33.680625 14750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:33.681382 14742 server_base.cc:1034] running on GCE node
I20260925 22:36:33.681761 14742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:36:33.682343 14742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:36:33.683497 14742 hybrid_clock.cc:648] HybridClock initialized: now 1790375793683484 us; error 28 us; skew 500 ppm
Sep 25 22:36:33 dist-test-slave-wjdr krb5kdc[14625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375793, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.246.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:36:33.686307 14742 init.cc:377] Logged in from keytab as oryx/127.6.246.193@KRBTEST.COM (short username oryx)
I20260925 22:36:33.687161 14742 webserver.cc:460] Webserver started at http://127.6.246.193:34047/ using document root <none> and password file <none>
I20260925 22:36:33.687724 14742 fs_manager.cc:359] Metadata directory not provided
I20260925 22:36:33.687794 14742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:36:33.688007 14742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:36:33.690045 14742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/minicluster-data/ts-0/data/instance:
uuid: "e35ce5c2bbc9406e86e15892c07a97fd"
format_stamp: "Formatted at 2026-09-25 22:36:33 on dist-test-slave-wjdr"
server_key: "9dae93062fd5fd98d9dc836e54ee9952"
server_key_iv: "1c0aa7ab31bc28ec590a37e09a957ebd"
server_key_version: "encryptionkey@0"
I20260925 22:36:33.690626 14742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/minicluster-data/ts-0/wal/instance:
uuid: "e35ce5c2bbc9406e86e15892c07a97fd"
format_stamp: "Formatted at 2026-09-25 22:36:33 on dist-test-slave-wjdr"
server_key: "9dae93062fd5fd98d9dc836e54ee9952"
server_key_iv: "1c0aa7ab31bc28ec590a37e09a957ebd"
server_key_version: "encryptionkey@0"
I20260925 22:36:33.694551 14742 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:36:33.696965 14757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:33.698117 14742 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:36:33.698241 14742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/minicluster-data/ts-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/minicluster-data/ts-0/wal
uuid: "e35ce5c2bbc9406e86e15892c07a97fd"
format_stamp: "Formatted at 2026-09-25 22:36:33 on dist-test-slave-wjdr"
server_key: "9dae93062fd5fd98d9dc836e54ee9952"
server_key_iv: "1c0aa7ab31bc28ec590a37e09a957ebd"
server_key_version: "encryptionkey@0"
I20260925 22:36:33.698347 14742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:36:33.718005 14742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:36:33.721205 14742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:36:33.721410 14742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:36:33.722810 14742 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:36:33.722877 14742 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:33.722945 14742 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:36:33.722997 14742 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 22:36:33 dist-test-slave-wjdr krb5kdc[14625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375793, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.246.193@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM
I20260925 22:36:33.733683 14742 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.193:38857
I20260925 22:36:33.734083 14872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.193:38857 every 8 connection(s)
I20260925 22:36:33.734661 14742 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
I20260925 22:36:33.736720 14719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:33.723519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.193:46721 (local address 127.6.246.254:38821)
0925 22:36:33.723660 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:33.723665 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:33.724534 (+ 869us) server_negotiation.cc:407] Connection header received
0925 22:36:33.725464 (+ 930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:33.725467 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:33.725524 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:33.725615 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:33.727274 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:33.727819 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:33.729772 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:33.730151 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:33.730280 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:33.733118 (+ 2838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:33.733122 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:33.733125 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:33.733155 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:33.734709 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:33.735332 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:33.735337 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:33.735340 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:33.735438 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:33.735774 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:33.735777 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:33.735778 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:33.735966 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:33.736040 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:33.736448 (+ 408us) server_negotiation.cc:299] Negotiation successful
0925 22:36:33.736603 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 22:36:33.737308 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 14742
I20260925 22:36:33.737439 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/minicluster-data/ts-0/wal/instance
I20260925 22:36:33.737587 14766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:33.723917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:38821 (local address 127.6.246.193:46721)
0925 22:36:33.724346 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:33.724403 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:33.725260 (+ 857us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:33.725834 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:33.725846 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:33.726369 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:33.727077 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:33.727099 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:33.727937 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:33.727944 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:33.729634 (+ 1690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:33.729650 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:33.730340 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:33.730348 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:33.730566 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:33.731138 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:33.731161 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:33.732948 (+ 1787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:36:33.734826 (+ 1878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:33.734831 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:33.734844 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:33.735200 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:33.735534 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:33.735537 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:33.735539 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:33.735687 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:33.736048 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:36:33.736052 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:36:33.736250 (+ 198us) client_negotiation.cc:769] Sending connection context
0925 22:36:33.736390 (+ 140us) client_negotiation.cc:240] Negotiation successful
0925 22:36:33.736672 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":124,"threads_started":1}
I20260925 22:36:33.737775 7131 external_mini_cluster.cc:1428] Setting key b784b92c05ffd7b2f3f6a9447ec4b378
I20260925 22:36:33.739199 14873 heartbeater.cc:347] Connected to a master server at 127.6.246.254:38821
I20260925 22:36:33.739337 14873 heartbeater.cc:464] Registering TS with master...
I20260925 22:36:33.739862 14873 heartbeater.cc:511] Master 127.6.246.254:38821 requested a full tablet report, sending...
I20260925 22:36:33.741415 14670 ts_manager.cc:194] Registered new tserver with Master: e35ce5c2bbc9406e86e15892c07a97fd (127.6.246.193:38857)
I20260925 22:36:33.742895 14670 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.246.193@KRBTEST.COM'} at 127.6.246.193:46721
WARNING: no policy specified for oryx/127.6.246.194@KRBTEST.COM; defaulting to no policy
I20260925 22:36:33.749018 14719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:33.741804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.193:54433 (local address 127.6.246.254:38821)
0925 22:36:33.741932 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:33.741939 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:33.741954 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:36:33.741990 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:33.741993 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:33.742031 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:33.742102 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:33.742594 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:33.743286 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:33.744422 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:33.744746 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:33.744820 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:33.745992 (+ 1172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:33.745995 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:33.745998 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:33.746019 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:33.747455 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:33.747879 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:33.747883 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:33.747884 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:33.747957 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:33.748282 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:33.748285 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:33.748286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:33.748519 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:33.748606 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:33.748814 (+ 208us) server_negotiation.cc:299] Negotiation successful
0925 22:36:33.748916 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 22:36:33.749001 14880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:33.741458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:38821 (local address 127.6.246.193:54433)
0925 22:36:33.741728 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:33.741744 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:33.741824 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:33.742145 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:33.742148 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:33.742360 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:33.742470 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:33.742476 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:33.743464 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:33.743467 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:33.744273 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:33.744283 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:33.744964 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:33.744967 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:33.745053 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:33.745535 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:33.745551 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:33.745854 (+ 303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:36:33.747566 (+ 1712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:33.747571 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:33.747573 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:33.747762 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:33.748050 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:33.748059 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:33.748061 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:33.748190 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:33.748602 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:36:33.748605 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:36:33.748676 (+ 71us) client_negotiation.cc:769] Sending connection context
0925 22:36:33.748758 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 22:36:33.748854 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":107,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.246.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.246.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/krb5kdc/oryx_127.6.246.194.keytab.
Entry for principal oryx/127.6.246.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/krb5kdc/oryx_127.6.246.194.keytab.
I20260925 22:36:33.763208 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.194:0
--local_ip_for_outbound_sockets=127.6.246.194
--webserver_interface=127.6.246.194
--webserver_port=0
--tserver_master_addrs=127.6.246.254:38821
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/krb5kdc/oryx_127.6.246.194.keytab
--principal=oryx/127.6.246.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:35249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:36:33.876031 14882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:36:33.876286 14882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:36:33.876330 14882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:36:33.880438 14882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:36:33.880553 14882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:36:33.880596 14882 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 22:36:33.880640 14882 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.194
I20260925 22:36:33.885773 14882 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:35249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/krb5kdc/oryx_127.6.246.194.keytab
--principal=oryx/127.6.246.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.246.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.246.254:38821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14882
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.194
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:36:33.886696 14882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:36:33.887650 14882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:36:33.896804 14887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:33.896811 14890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:33.896891 14882 server_base.cc:1034] running on GCE node
W20260925 22:36:33.896818 14888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:33.897423 14882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:36:33.898005 14882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:36:33.899151 14882 hybrid_clock.cc:648] HybridClock initialized: now 1790375793899139 us; error 25 us; skew 500 ppm
Sep 25 22:36:33 dist-test-slave-wjdr krb5kdc[14625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375793, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.246.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:36:33.901875 14882 init.cc:377] Logged in from keytab as oryx/127.6.246.194@KRBTEST.COM (short username oryx)
I20260925 22:36:33.902757 14882 webserver.cc:460] Webserver started at http://127.6.246.194:45117/ using document root <none> and password file <none>
I20260925 22:36:33.903290 14882 fs_manager.cc:359] Metadata directory not provided
I20260925 22:36:33.903395 14882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:36:33.903605 14882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:36:33.905592 14882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/minicluster-data/ts-1/data/instance:
uuid: "063799ab6b434788be1447722615f67d"
format_stamp: "Formatted at 2026-09-25 22:36:33 on dist-test-slave-wjdr"
server_key: "32f56f7e3eef739c48bdc0c76870a0c6"
server_key_iv: "9f00ab6d12a8262d89e895fd40c39a4f"
server_key_version: "encryptionkey@0"
I20260925 22:36:33.906152 14882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/minicluster-data/ts-1/wal/instance:
uuid: "063799ab6b434788be1447722615f67d"
format_stamp: "Formatted at 2026-09-25 22:36:33 on dist-test-slave-wjdr"
server_key: "32f56f7e3eef739c48bdc0c76870a0c6"
server_key_iv: "9f00ab6d12a8262d89e895fd40c39a4f"
server_key_version: "encryptionkey@0"
I20260925 22:36:33.909817 14882 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:36:33.912101 14897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:33.913398 14882 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:36:33.913512 14882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/minicluster-data/ts-1/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/minicluster-data/ts-1/wal
uuid: "063799ab6b434788be1447722615f67d"
format_stamp: "Formatted at 2026-09-25 22:36:33 on dist-test-slave-wjdr"
server_key: "32f56f7e3eef739c48bdc0c76870a0c6"
server_key_iv: "9f00ab6d12a8262d89e895fd40c39a4f"
server_key_version: "encryptionkey@0"
I20260925 22:36:33.913619 14882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:36:33.927260 14882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:36:33.930450 14882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:36:33.930624 14882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:36:33.931900 14882 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:36:33.931967 14882 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:33.932029 14882 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:36:33.932080 14882 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 22:36:33 dist-test-slave-wjdr krb5kdc[14625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375793, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.246.194@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM
I20260925 22:36:33.942240 14882 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.194:36953
I20260925 22:36:33.942296 15012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.194:36953 every 8 connection(s)
I20260925 22:36:33.943372 14882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
I20260925 22:36:33.945806 14719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:33.932840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.194:60859 (local address 127.6.246.254:38821)
0925 22:36:33.933007 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:33.933012 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:33.933858 (+ 846us) server_negotiation.cc:407] Connection header received
0925 22:36:33.934984 (+ 1126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:33.934988 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:33.935052 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:33.935156 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:33.936595 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:33.937478 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:33.939208 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:33.939542 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:33.939648 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:33.942482 (+ 2834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:33.942486 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:33.942489 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:33.942517 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:33.944045 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:33.944676 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:33.944679 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:33.944680 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:33.944758 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:33.944989 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:33.944991 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:33.944992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:33.945144 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:33.945213 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:33.945594 (+ 381us) server_negotiation.cc:299] Negotiation successful
0925 22:36:33.945692 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 22:36:33.946543 14906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:33.933248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:38821 (local address 127.6.246.194:60859)
0925 22:36:33.933672 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:33.933720 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:33.934742 (+ 1022us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:33.935360 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:33.935371 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:33.935749 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:33.936422 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:33.936445 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:33.937613 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:33.937618 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:33.939083 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:33.939096 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:33.939699 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:33.939703 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:33.939896 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:33.940542 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:33.940565 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:33.942303 (+ 1738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:36:33.944185 (+ 1882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:33.944191 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:33.944203 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:33.944577 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:33.944820 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:33.944822 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:33.944823 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:33.944933 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:33.945203 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:36:33.945207 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:36:33.945404 (+ 197us) client_negotiation.cc:769] Sending connection context
0925 22:36:33.945535 (+ 131us) client_negotiation.cc:240] Negotiation successful
0925 22:36:33.945754 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":126,"threads_started":1}
I20260925 22:36:33.947614 15013 heartbeater.cc:347] Connected to a master server at 127.6.246.254:38821
I20260925 22:36:33.947741 15013 heartbeater.cc:464] Registering TS with master...
I20260925 22:36:33.948263 15013 heartbeater.cc:511] Master 127.6.246.254:38821 requested a full tablet report, sending...
I20260925 22:36:33.949291 14670 ts_manager.cc:194] Registered new tserver with Master: 063799ab6b434788be1447722615f67d (127.6.246.194:36953)
I20260925 22:36:33.949782 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 14882
I20260925 22:36:33.949910 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/minicluster-data/ts-1/wal/instance
I20260925 22:36:33.950016 14670 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.246.194@KRBTEST.COM'} at 127.6.246.194:60859
I20260925 22:36:33.950268 7131 external_mini_cluster.cc:1428] Setting key 18df455414c559b66297eaed425a8aec
I20260925 22:36:33.957547 14719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:33.949911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.194:39613 (local address 127.6.246.254:38821)
0925 22:36:33.950022 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:33.950026 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:33.950255 (+ 229us) server_negotiation.cc:407] Connection header received
0925 22:36:33.950390 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:33.950393 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:33.950435 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:33.950513 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:33.951023 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:33.951745 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:33.952838 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:33.953040 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:33.953119 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:33.954407 (+ 1288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:33.954413 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:33.954418 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:33.954447 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:33.955850 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:33.956324 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:33.956327 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:33.956328 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:33.956393 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:33.956746 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:33.956749 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:33.956751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:33.956987 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:33.957066 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:33.957360 (+ 294us) server_negotiation.cc:299] Negotiation successful
0925 22:36:33.957444 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 22:36:33.957742 15019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:33.949817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:38821 (local address 127.6.246.194:39613)
0925 22:36:33.950180 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:33.950195 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:33.950294 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:33.950547 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:33.950550 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:33.950755 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:33.950887 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:33.950894 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:33.951917 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:33.951920 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:33.952723 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:33.952738 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:33.953151 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:33.953158 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:33.953285 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:33.953817 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:33.953837 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:33.954282 (+ 445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:36:33.955952 (+ 1670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:33.955958 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:33.955963 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:33.956218 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:33.956508 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:33.956514 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:33.956519 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:33.956638 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:33.957088 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:36:33.957093 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:36:33.957196 (+ 103us) client_negotiation.cc:769] Sending connection context
0925 22:36:33.957313 (+ 117us) client_negotiation.cc:240] Negotiation successful
0925 22:36:33.957470 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":83,"threads_started":1}
WARNING: no policy specified for oryx/127.6.246.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.246.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.246.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/krb5kdc/oryx_127.6.246.195.keytab.
Entry for principal oryx/127.6.246.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/krb5kdc/oryx_127.6.246.195.keytab.
I20260925 22:36:33.979035 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.195:0
--local_ip_for_outbound_sockets=127.6.246.195
--webserver_interface=127.6.246.195
--webserver_port=0
--tserver_master_addrs=127.6.246.254:38821
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/krb5kdc/oryx_127.6.246.195.keytab
--principal=oryx/127.6.246.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:35249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:36:34.091382 15022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:36:34.091646 15022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:36:34.091689 15022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:36:34.095769 15022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:36:34.095821 15022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:36:34.095890 15022 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 22:36:34.095970 15022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.195
I20260925 22:36:34.101068 15022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:35249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/krb5kdc/oryx_127.6.246.195.keytab
--principal=oryx/127.6.246.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.246.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.246.254:38821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.195
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:36:34.102072 15022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:36:34.103052 15022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 22:36:34.112227 15022 server_base.cc:1034] running on GCE node
W20260925 22:36:34.112217 15028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:34.112181 15027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:34.112181 15030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:34.112802 15022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:36:34.113401 15022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:36:34.114586 15022 hybrid_clock.cc:648] HybridClock initialized: now 1790375794114572 us; error 26 us; skew 500 ppm
Sep 25 22:36:34 dist-test-slave-wjdr krb5kdc[14625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375794, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.246.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:36:34.117272 15022 init.cc:377] Logged in from keytab as oryx/127.6.246.195@KRBTEST.COM (short username oryx)
I20260925 22:36:34.118139 15022 webserver.cc:460] Webserver started at http://127.6.246.195:32941/ using document root <none> and password file <none>
I20260925 22:36:34.118660 15022 fs_manager.cc:359] Metadata directory not provided
I20260925 22:36:34.118743 15022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:36:34.118956 15022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:36:34.120903 15022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/minicluster-data/ts-2/data/instance:
uuid: "c99f2a7a833f49c68022c8a09e09c9ef"
format_stamp: "Formatted at 2026-09-25 22:36:34 on dist-test-slave-wjdr"
server_key: "0c312c041e6ec033ad7cc4b5d0f39e9b"
server_key_iv: "ba5388af8d6bb38b28f1bcfc7ab12d94"
server_key_version: "encryptionkey@0"
I20260925 22:36:34.121424 15022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/minicluster-data/ts-2/wal/instance:
uuid: "c99f2a7a833f49c68022c8a09e09c9ef"
format_stamp: "Formatted at 2026-09-25 22:36:34 on dist-test-slave-wjdr"
server_key: "0c312c041e6ec033ad7cc4b5d0f39e9b"
server_key_iv: "ba5388af8d6bb38b28f1bcfc7ab12d94"
server_key_version: "encryptionkey@0"
I20260925 22:36:34.125370 15022 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 22:36:34.127588 15037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:34.128780 15022 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 22:36:34.128911 15022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/minicluster-data/ts-2/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/minicluster-data/ts-2/wal
uuid: "c99f2a7a833f49c68022c8a09e09c9ef"
format_stamp: "Formatted at 2026-09-25 22:36:34 on dist-test-slave-wjdr"
server_key: "0c312c041e6ec033ad7cc4b5d0f39e9b"
server_key_iv: "ba5388af8d6bb38b28f1bcfc7ab12d94"
server_key_version: "encryptionkey@0"
I20260925 22:36:34.129040 15022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:36:34.147222 15022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:36:34.150337 15022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:36:34.150523 15022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:36:34.151829 15022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:36:34.151895 15022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:34.151958 15022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:36:34.152005 15022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 22:36:34 dist-test-slave-wjdr krb5kdc[14625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375794, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.246.195@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM
I20260925 22:36:34.162559 15022 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.195:44587
I20260925 22:36:34.162850 15152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.195:44587 every 8 connection(s)
I20260925 22:36:34.163671 15022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
I20260925 22:36:34.164724 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 15022
I20260925 22:36:34.164885 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/minicluster-data/ts-2/wal/instance
I20260925 22:36:34.165230 7131 external_mini_cluster.cc:1428] Setting key 261b062e3444ea198756ee9ffad9b4b1
I20260925 22:36:34.165817 14719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:34.152677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.195:56305 (local address 127.6.246.254:38821)
0925 22:36:34.152824 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:34.152829 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:34.153622 (+ 793us) server_negotiation.cc:407] Connection header received
0925 22:36:34.154642 (+ 1020us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:34.154645 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:34.154698 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:34.154753 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:34.156437 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:34.157347 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:34.158856 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:34.159211 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:34.159325 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:34.161772 (+ 2447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:34.161776 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:34.161779 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:34.161812 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:34.163533 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:34.164106 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:34.164109 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:34.164111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:34.164190 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:34.164665 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:34.164668 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:34.164670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:34.164987 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:34.165096 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:34.165590 (+ 494us) server_negotiation.cc:299] Negotiation successful
0925 22:36:34.165696 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 22:36:34.166682 15046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:34.153031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:38821 (local address 127.6.246.195:56305)
0925 22:36:34.153476 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:34.153524 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:34.154383 (+ 859us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:34.155168 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:34.155180 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:34.155645 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:34.156242 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:34.156262 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:34.157493 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:34.157497 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:34.158725 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:34.158734 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:34.159370 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:34.159374 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:34.159547 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:34.160094 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:34.160110 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:34.161618 (+ 1508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:36:34.163642 (+ 2024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:34.163646 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:34.163656 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:34.164002 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:34.164296 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:34.164299 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:34.164301 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:34.164531 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:34.165086 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:36:34.165091 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:36:34.165329 (+ 238us) client_negotiation.cc:769] Sending connection context
0925 22:36:34.165496 (+ 167us) client_negotiation.cc:240] Negotiation successful
0925 22:36:34.165832 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":117,"threads_started":1}
I20260925 22:36:34.167855 15153 heartbeater.cc:347] Connected to a master server at 127.6.246.254:38821
I20260925 22:36:34.168005 15153 heartbeater.cc:464] Registering TS with master...
I20260925 22:36:34.168504 15153 heartbeater.cc:511] Master 127.6.246.254:38821 requested a full tablet report, sending...
I20260925 22:36:34.169396 14670 ts_manager.cc:194] Registered new tserver with Master: c99f2a7a833f49c68022c8a09e09c9ef (127.6.246.195:44587)
I20260925 22:36:34.170001 14670 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.246.195@KRBTEST.COM'} at 127.6.246.195:56305
I20260925 22:36:34.176093 15159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:34.169870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:38821 (local address 127.6.246.195:43897)
0925 22:36:34.170115 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:34.170131 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:34.170236 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:34.170475 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:34.170477 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:34.170680 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:34.170823 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:34.170831 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:34.171548 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:34.171551 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:34.172219 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:34.172228 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:34.172789 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:34.172793 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:34.172868 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:34.173343 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:34.173361 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:34.173637 (+ 276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:36:34.174993 (+ 1356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:34.174995 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:34.174997 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:34.175167 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:34.175385 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:34.175387 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:34.175388 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:34.175453 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:34.175735 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:36:34.175737 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:36:34.175805 (+ 68us) client_negotiation.cc:769] Sending connection context
0925 22:36:34.175880 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 22:36:34.175975 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":25,"thread_start_us":93,"threads_started":1}
I20260925 22:36:34.176108 14719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:34.169924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.195:43897 (local address 127.6.246.254:38821)
0925 22:36:34.170037 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:34.170041 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:34.170200 (+ 159us) server_negotiation.cc:407] Connection header received
0925 22:36:34.170328 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:34.170330 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:34.170372 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:34.170457 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:34.170935 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:34.171429 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:34.172387 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:34.172683 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:34.172771 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:34.173741 (+ 970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:34.173744 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:34.173746 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:34.173780 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:34.174904 (+ 1124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:34.175243 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:34.175245 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:34.175246 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:34.175310 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:34.175521 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:34.175523 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:34.175524 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:34.175666 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:34.175737 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:34.175932 (+ 195us) server_negotiation.cc:299] Negotiation successful
0925 22:36:34.176012 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 22:36:34.178988 7131 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 22:36:34 dist-test-slave-wjdr krb5kdc[14625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375794, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 22:36:34 dist-test-slave-wjdr krb5kdc[14625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:36:34 dist-test-slave-wjdr krb5kdc[14625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
W20260925 22:36:34.201706 14719 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:36:34.195637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50312 (local address 127.6.246.254:38821)
0925 22:36:34.195757 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:34.195761 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:34.195772 (+ 11us) server_negotiation.cc:407] Connection header received
0925 22:36:34.195811 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:34.195813 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:34.195851 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:34.195942 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:34.196687 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:34.197202 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:34.198051 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:34.198272 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:34.198400 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:34.201539 (+ 3139us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50312: BlockingRecv error: recv got EOF from 127.0.0.1:50312 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 25 22:36:34 dist-test-slave-wjdr krb5kdc[14625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375794, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM
I20260925 22:36:34.211066 14719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:34.203234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50324 (local address 127.6.246.254:38821)
0925 22:36:34.203391 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:34.203395 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:34.203489 (+ 94us) server_negotiation.cc:407] Connection header received
0925 22:36:34.203623 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:34.203626 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:34.203670 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:34.203750 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:34.204342 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:34.204904 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:34.205662 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:34.205880 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:34.205982 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:34.208162 (+ 2180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:34.208166 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:34.208168 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:34.208197 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:34.209424 (+ 1227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:34.209835 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:34.209839 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:34.209858 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:34.209951 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:34.210223 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:34.210228 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:34.210229 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:34.210402 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:34.210532 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:34.210772 (+ 240us) server_negotiation.cc:299] Negotiation successful
0925 22:36:34.210903 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 22:36:34.213807 14670 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50324:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:36:34.216811 14670 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:36:34.230863 15180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:34.225861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54576 (local address 127.6.246.195:44587)
0925 22:36:34.226285 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:34.226296 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:34.226319 (+ 23us) server_negotiation.cc:407] Connection header received
0925 22:36:34.226396 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:34.226401 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:34.226578 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:34.226719 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:36:34.227048 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:34.227779 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:34.229405 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:34.230159 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:34.230294 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:34.230479 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:34.230610 (+ 131us) server_negotiation.cc:299] Negotiation successful
0925 22:36:34.230706 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":111,"threads_started":1}
I20260925 22:36:34.230850 15175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:34.225460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.195:44587 (local address 127.0.0.1:54576)
0925 22:36:34.226224 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:34.226236 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:34.226303 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:34.226769 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:34.226771 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:34.226786 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:36:34.226921 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:34.226931 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:34.227902 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:34.227906 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:34.229274 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:34.229284 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:34.230294 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:34.230298 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:34.230409 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:34.230448 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 22:36:34.230573 (+ 125us) client_negotiation.cc:240] Negotiation successful
0925 22:36:34.230698 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":709,"thread_start_us":96,"threads_started":1}
I20260925 22:36:34.231523 15177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:34.224973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58542 (local address 127.6.246.193:38857)
0925 22:36:34.225465 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:34.225482 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:34.226119 (+ 637us) server_negotiation.cc:407] Connection header received
0925 22:36:34.226210 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:34.226215 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:34.226431 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:34.226597 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:36:34.227096 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:34.228301 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:34.229730 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:34.230540 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:34.230664 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:34.230831 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:34.231252 (+ 421us) server_negotiation.cc:299] Negotiation successful
0925 22:36:34.231362 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":91,"threads_started":1}
I20260925 22:36:34.231984 15178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:34.225351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45972 (local address 127.6.246.194:36953)
0925 22:36:34.225747 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:34.225760 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:34.226125 (+ 365us) server_negotiation.cc:407] Connection header received
0925 22:36:34.226209 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:34.226215 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:34.226431 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:34.226597 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:36:34.226876 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:34.228220 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:34.229842 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:34.230593 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:34.230715 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:34.230841 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:34.231717 (+ 876us) server_negotiation.cc:299] Negotiation successful
0925 22:36:34.231835 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":101,"threads_started":1}
I20260925 22:36:34.232072 15179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:34.224960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.194:36953 (local address 127.0.0.1:45972)
0925 22:36:34.225952 (+ 992us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:34.225969 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:34.226064 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:34.226595 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:34.226601 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:34.226623 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:36:34.226760 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:34.226767 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:34.228385 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:34.228390 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:34.229648 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:34.229656 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:34.230892 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:34.230896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:34.231056 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:34.231139 (+ 83us) client_negotiation.cc:769] Sending connection context
0925 22:36:34.231210 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 22:36:34.231328 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":926,"mutex_wait_us":146,"thread_start_us":77,"threads_started":1}
I20260925 22:36:34.232240 15176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:34.224771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.193:38857 (local address 127.0.0.1:58542)
0925 22:36:34.225728 (+ 957us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:34.225781 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:34.225960 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:34.226864 (+ 904us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:34.226866 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:34.226892 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:36:34.226959 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:34.226963 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:34.228429 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:34.228432 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:34.229593 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:34.229604 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:34.230897 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:34.230900 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:34.231080 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:34.231137 (+ 57us) client_negotiation.cc:769] Sending connection context
0925 22:36:34.231210 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 22:36:34.232044 (+ 834us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":779,"mutex_wait_us":134,"thread_start_us":90,"threads_started":1}
I20260925 22:36:34.234148 15087 tablet_service.cc:1467] Processing CreateTablet for tablet fef6b640e9704c899e5e0731e58c5306 (DEFAULT_TABLE table=test-table [id=ca2f52055e514f10adc76fc6f08e88c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:36:34.234630 14947 tablet_service.cc:1467] Processing CreateTablet for tablet fef6b640e9704c899e5e0731e58c5306 (DEFAULT_TABLE table=test-table [id=ca2f52055e514f10adc76fc6f08e88c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:36:34.235152 15087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fef6b640e9704c899e5e0731e58c5306. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:36:34.235510 14947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fef6b640e9704c899e5e0731e58c5306. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:36:34.238554 14807 tablet_service.cc:1467] Processing CreateTablet for tablet fef6b640e9704c899e5e0731e58c5306 (DEFAULT_TABLE table=test-table [id=ca2f52055e514f10adc76fc6f08e88c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:36:34.239643 14807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fef6b640e9704c899e5e0731e58c5306. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:36:34.242355 15181 tablet_bootstrap.cc:492] T fef6b640e9704c899e5e0731e58c5306 P c99f2a7a833f49c68022c8a09e09c9ef: Bootstrap starting.
I20260925 22:36:34.244408 15181 tablet_bootstrap.cc:654] T fef6b640e9704c899e5e0731e58c5306 P c99f2a7a833f49c68022c8a09e09c9ef: Neither blocks nor log segments found. Creating new log.
I20260925 22:36:34.245169 15181 log.cc:824] T fef6b640e9704c899e5e0731e58c5306 P c99f2a7a833f49c68022c8a09e09c9ef: Log is configured to *not* fsync() on all Append() calls
I20260925 22:36:34.246222 15184 tablet_bootstrap.cc:492] T fef6b640e9704c899e5e0731e58c5306 P e35ce5c2bbc9406e86e15892c07a97fd: Bootstrap starting.
I20260925 22:36:34.246809 15182 tablet_bootstrap.cc:492] T fef6b640e9704c899e5e0731e58c5306 P 063799ab6b434788be1447722615f67d: Bootstrap starting.
I20260925 22:36:34.247187 15181 tablet_bootstrap.cc:492] T fef6b640e9704c899e5e0731e58c5306 P c99f2a7a833f49c68022c8a09e09c9ef: No bootstrap required, opened a new log
I20260925 22:36:34.247393 15181 ts_tablet_manager.cc:1397] T fef6b640e9704c899e5e0731e58c5306 P c99f2a7a833f49c68022c8a09e09c9ef: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260925 22:36:34.248888 15182 tablet_bootstrap.cc:654] T fef6b640e9704c899e5e0731e58c5306 P 063799ab6b434788be1447722615f67d: Neither blocks nor log segments found. Creating new log.
I20260925 22:36:34.249133 15184 tablet_bootstrap.cc:654] T fef6b640e9704c899e5e0731e58c5306 P e35ce5c2bbc9406e86e15892c07a97fd: Neither blocks nor log segments found. Creating new log.
I20260925 22:36:34.249708 15182 log.cc:824] T fef6b640e9704c899e5e0731e58c5306 P 063799ab6b434788be1447722615f67d: Log is configured to *not* fsync() on all Append() calls
I20260925 22:36:34.249857 15184 log.cc:824] T fef6b640e9704c899e5e0731e58c5306 P e35ce5c2bbc9406e86e15892c07a97fd: Log is configured to *not* fsync() on all Append() calls
I20260925 22:36:34.251385 15181 raft_consensus.cc:348] T fef6b640e9704c899e5e0731e58c5306 P c99f2a7a833f49c68022c8a09e09c9ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35ce5c2bbc9406e86e15892c07a97fd" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 38857 } } peers { permanent_uuid: "063799ab6b434788be1447722615f67d" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 36953 } } peers { permanent_uuid: "c99f2a7a833f49c68022c8a09e09c9ef" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 44587 } }
I20260925 22:36:34.251617 15184 tablet_bootstrap.cc:492] T fef6b640e9704c899e5e0731e58c5306 P e35ce5c2bbc9406e86e15892c07a97fd: No bootstrap required, opened a new log
I20260925 22:36:34.251623 15181 raft_consensus.cc:374] T fef6b640e9704c899e5e0731e58c5306 P c99f2a7a833f49c68022c8a09e09c9ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:36:34.251642 15182 tablet_bootstrap.cc:492] T fef6b640e9704c899e5e0731e58c5306 P 063799ab6b434788be1447722615f67d: No bootstrap required, opened a new log
I20260925 22:36:34.251685 15181 raft_consensus.cc:729] T fef6b640e9704c899e5e0731e58c5306 P c99f2a7a833f49c68022c8a09e09c9ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c99f2a7a833f49c68022c8a09e09c9ef, State: Initialized, Role: FOLLOWER
I20260925 22:36:34.251792 15182 ts_tablet_manager.cc:1397] T fef6b640e9704c899e5e0731e58c5306 P 063799ab6b434788be1447722615f67d: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.005s
I20260925 22:36:34.251792 15184 ts_tablet_manager.cc:1397] T fef6b640e9704c899e5e0731e58c5306 P e35ce5c2bbc9406e86e15892c07a97fd: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 22:36:34.252140 15181 consensus_queue.cc:260] T fef6b640e9704c899e5e0731e58c5306 P c99f2a7a833f49c68022c8a09e09c9ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35ce5c2bbc9406e86e15892c07a97fd" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 38857 } } peers { permanent_uuid: "063799ab6b434788be1447722615f67d" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 36953 } } peers { permanent_uuid: "c99f2a7a833f49c68022c8a09e09c9ef" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 44587 } }
I20260925 22:36:34.253101 15181 ts_tablet_manager.cc:1428] T fef6b640e9704c899e5e0731e58c5306 P c99f2a7a833f49c68022c8a09e09c9ef: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260925 22:36:34.255657 15182 raft_consensus.cc:348] T fef6b640e9704c899e5e0731e58c5306 P 063799ab6b434788be1447722615f67d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35ce5c2bbc9406e86e15892c07a97fd" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 38857 } } peers { permanent_uuid: "063799ab6b434788be1447722615f67d" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 36953 } } peers { permanent_uuid: "c99f2a7a833f49c68022c8a09e09c9ef" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 44587 } }
I20260925 22:36:34.255657 15184 raft_consensus.cc:348] T fef6b640e9704c899e5e0731e58c5306 P e35ce5c2bbc9406e86e15892c07a97fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35ce5c2bbc9406e86e15892c07a97fd" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 38857 } } peers { permanent_uuid: "063799ab6b434788be1447722615f67d" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 36953 } } peers { permanent_uuid: "c99f2a7a833f49c68022c8a09e09c9ef" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 44587 } }
I20260925 22:36:34.255921 15184 raft_consensus.cc:374] T fef6b640e9704c899e5e0731e58c5306 P e35ce5c2bbc9406e86e15892c07a97fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:36:34.255921 15182 raft_consensus.cc:374] T fef6b640e9704c899e5e0731e58c5306 P 063799ab6b434788be1447722615f67d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:36:34.255973 15184 raft_consensus.cc:729] T fef6b640e9704c899e5e0731e58c5306 P e35ce5c2bbc9406e86e15892c07a97fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e35ce5c2bbc9406e86e15892c07a97fd, State: Initialized, Role: FOLLOWER
I20260925 22:36:34.255973 15182 raft_consensus.cc:729] T fef6b640e9704c899e5e0731e58c5306 P 063799ab6b434788be1447722615f67d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 063799ab6b434788be1447722615f67d, State: Initialized, Role: FOLLOWER
I20260925 22:36:34.256402 15184 consensus_queue.cc:260] T fef6b640e9704c899e5e0731e58c5306 P e35ce5c2bbc9406e86e15892c07a97fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35ce5c2bbc9406e86e15892c07a97fd" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 38857 } } peers { permanent_uuid: "063799ab6b434788be1447722615f67d" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 36953 } } peers { permanent_uuid: "c99f2a7a833f49c68022c8a09e09c9ef" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 44587 } }
I20260925 22:36:34.256651 15153 heartbeater.cc:503] Master 127.6.246.254:38821 was elected leader, sending a full tablet report...
I20260925 22:36:34.256397 15182 consensus_queue.cc:260] T fef6b640e9704c899e5e0731e58c5306 P 063799ab6b434788be1447722615f67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35ce5c2bbc9406e86e15892c07a97fd" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 38857 } } peers { permanent_uuid: "063799ab6b434788be1447722615f67d" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 36953 } } peers { permanent_uuid: "c99f2a7a833f49c68022c8a09e09c9ef" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 44587 } }
I20260925 22:36:34.257336 15184 ts_tablet_manager.cc:1428] T fef6b640e9704c899e5e0731e58c5306 P e35ce5c2bbc9406e86e15892c07a97fd: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260925 22:36:34.257450 15182 ts_tablet_manager.cc:1428] T fef6b640e9704c899e5e0731e58c5306 P 063799ab6b434788be1447722615f67d: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260925 22:36:34.257735 15013 heartbeater.cc:503] Master 127.6.246.254:38821 was elected leader, sending a full tablet report...
I20260925 22:36:34.257764 14873 heartbeater.cc:503] Master 127.6.246.254:38821 was elected leader, sending a full tablet report...
I20260925 22:36:34.366907 15189 raft_consensus.cc:482] T fef6b640e9704c899e5e0731e58c5306 P 063799ab6b434788be1447722615f67d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:36:34.367137 15189 raft_consensus.cc:504] T fef6b640e9704c899e5e0731e58c5306 P 063799ab6b434788be1447722615f67d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35ce5c2bbc9406e86e15892c07a97fd" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 38857 } } peers { permanent_uuid: "063799ab6b434788be1447722615f67d" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 36953 } } peers { permanent_uuid: "c99f2a7a833f49c68022c8a09e09c9ef" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 44587 } }
I20260925 22:36:34.368409 15189 leader_election.cc:288] T fef6b640e9704c899e5e0731e58c5306 P 063799ab6b434788be1447722615f67d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e35ce5c2bbc9406e86e15892c07a97fd (127.6.246.193:38857), c99f2a7a833f49c68022c8a09e09c9ef (127.6.246.195:44587)
I20260925 22:36:34.372143 14906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:34.368745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.193:38857 (local address 127.6.246.194:36877)
0925 22:36:34.368906 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:34.368924 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:34.369018 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:34.369395 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:34.369398 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:34.369421 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:36:34.369514 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:34.369519 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:34.370261 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:34.370266 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:34.371075 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:34.371083 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:34.371717 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:34.371720 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:34.371840 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:34.371857 (+ 17us) client_negotiation.cc:769] Sending connection context
0925 22:36:34.371943 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 22:36:34.372013 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 22:36:34.372143 15177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:34.368821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.194:36877 (local address 127.6.246.193:38857)
0925 22:36:34.368963 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:34.368967 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:34.369001 (+ 34us) server_negotiation.cc:407] Connection header received
0925 22:36:34.369136 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:34.369142 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:34.369193 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:34.369283 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:36:34.369629 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:34.370148 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:34.371204 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:34.371604 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:34.371728 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:34.371784 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:34.371959 (+ 175us) server_negotiation.cc:299] Negotiation successful
0925 22:36:34.372028 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 22:36:34.372849 14827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fef6b640e9704c899e5e0731e58c5306" candidate_uuid: "063799ab6b434788be1447722615f67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e35ce5c2bbc9406e86e15892c07a97fd" is_pre_election: true
I20260925 22:36:34.372931 15180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:34.369190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.194:45321 (local address 127.6.246.195:44587)
0925 22:36:34.369309 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:34.369314 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:34.369485 (+ 171us) server_negotiation.cc:407] Connection header received
0925 22:36:34.369620 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:34.369623 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:34.369669 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:34.369755 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:36:34.370027 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:34.370549 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:34.371899 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:34.372400 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:34.372558 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:34.372613 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:34.372787 (+ 174us) server_negotiation.cc:299] Negotiation successful
0925 22:36:34.372838 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 22:36:34.372970 15190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:34.369026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.195:44587 (local address 127.6.246.194:45321)
0925 22:36:34.369393 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:34.369405 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:34.369513 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:34.369766 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:34.369769 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:34.369782 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:36:34.369918 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:34.369923 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:34.370657 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:34.370663 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:34.371767 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:34.371775 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:34.372564 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:34.372569 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:34.372662 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:34.372683 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 22:36:34.372755 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 22:36:34.372815 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":190,"threads_started":1}
I20260925 22:36:34.373165 14827 raft_consensus.cc:2454] T fef6b640e9704c899e5e0731e58c5306 P e35ce5c2bbc9406e86e15892c07a97fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 063799ab6b434788be1447722615f67d in term 0.
I20260925 22:36:34.373617 14898 leader_election.cc:302] T fef6b640e9704c899e5e0731e58c5306 P 063799ab6b434788be1447722615f67d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 063799ab6b434788be1447722615f67d, e35ce5c2bbc9406e86e15892c07a97fd; no voters:
I20260925 22:36:34.373904 15189 raft_consensus.cc:2781] T fef6b640e9704c899e5e0731e58c5306 P 063799ab6b434788be1447722615f67d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:36:34.373886 15107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fef6b640e9704c899e5e0731e58c5306" candidate_uuid: "063799ab6b434788be1447722615f67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c99f2a7a833f49c68022c8a09e09c9ef" is_pre_election: true
I20260925 22:36:34.373996 15189 raft_consensus.cc:482] T fef6b640e9704c899e5e0731e58c5306 P 063799ab6b434788be1447722615f67d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:36:34.374042 15189 raft_consensus.cc:3037] T fef6b640e9704c899e5e0731e58c5306 P 063799ab6b434788be1447722615f67d [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:36:34.374150 15107 raft_consensus.cc:2454] T fef6b640e9704c899e5e0731e58c5306 P c99f2a7a833f49c68022c8a09e09c9ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 063799ab6b434788be1447722615f67d in term 0.
I20260925 22:36:34.375098 15189 raft_consensus.cc:504] T fef6b640e9704c899e5e0731e58c5306 P 063799ab6b434788be1447722615f67d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35ce5c2bbc9406e86e15892c07a97fd" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 38857 } } peers { permanent_uuid: "063799ab6b434788be1447722615f67d" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 36953 } } peers { permanent_uuid: "c99f2a7a833f49c68022c8a09e09c9ef" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 44587 } }
I20260925 22:36:34.375376 15189 leader_election.cc:288] T fef6b640e9704c899e5e0731e58c5306 P 063799ab6b434788be1447722615f67d [CANDIDATE]: Term 1 election: Requested vote from peers e35ce5c2bbc9406e86e15892c07a97fd (127.6.246.193:38857), c99f2a7a833f49c68022c8a09e09c9ef (127.6.246.195:44587)
I20260925 22:36:34.375735 14827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fef6b640e9704c899e5e0731e58c5306" candidate_uuid: "063799ab6b434788be1447722615f67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e35ce5c2bbc9406e86e15892c07a97fd"
I20260925 22:36:34.375758 15107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fef6b640e9704c899e5e0731e58c5306" candidate_uuid: "063799ab6b434788be1447722615f67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c99f2a7a833f49c68022c8a09e09c9ef"
I20260925 22:36:34.375864 14827 raft_consensus.cc:3037] T fef6b640e9704c899e5e0731e58c5306 P e35ce5c2bbc9406e86e15892c07a97fd [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:36:34.375851 15107 raft_consensus.cc:3037] T fef6b640e9704c899e5e0731e58c5306 P c99f2a7a833f49c68022c8a09e09c9ef [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:36:34.376847 14827 raft_consensus.cc:2454] T fef6b640e9704c899e5e0731e58c5306 P e35ce5c2bbc9406e86e15892c07a97fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 063799ab6b434788be1447722615f67d in term 1.
I20260925 22:36:34.376988 15107 raft_consensus.cc:2454] T fef6b640e9704c899e5e0731e58c5306 P c99f2a7a833f49c68022c8a09e09c9ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 063799ab6b434788be1447722615f67d in term 1.
I20260925 22:36:34.377183 14898 leader_election.cc:302] T fef6b640e9704c899e5e0731e58c5306 P 063799ab6b434788be1447722615f67d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 063799ab6b434788be1447722615f67d, e35ce5c2bbc9406e86e15892c07a97fd; no voters:
I20260925 22:36:34.377337 15189 raft_consensus.cc:2781] T fef6b640e9704c899e5e0731e58c5306 P 063799ab6b434788be1447722615f67d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:36:34.377574 15189 raft_consensus.cc:686] T fef6b640e9704c899e5e0731e58c5306 P 063799ab6b434788be1447722615f67d [term 1 LEADER]: Becoming Leader. State: Replica: 063799ab6b434788be1447722615f67d, State: Running, Role: LEADER
I20260925 22:36:34.377939 15189 consensus_queue.cc:237] T fef6b640e9704c899e5e0731e58c5306 P 063799ab6b434788be1447722615f67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35ce5c2bbc9406e86e15892c07a97fd" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 38857 } } peers { permanent_uuid: "063799ab6b434788be1447722615f67d" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 36953 } } peers { permanent_uuid: "c99f2a7a833f49c68022c8a09e09c9ef" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 44587 } }
I20260925 22:36:34.381114 14670 catalog_manager.cc:5495] T fef6b640e9704c899e5e0731e58c5306 P 063799ab6b434788be1447722615f67d reported cstate change: term changed from 0 to 1, leader changed from <none> to 063799ab6b434788be1447722615f67d (127.6.246.194). New cstate: current_term: 1 leader_uuid: "063799ab6b434788be1447722615f67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35ce5c2bbc9406e86e15892c07a97fd" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 38857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "063799ab6b434788be1447722615f67d" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 36953 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c99f2a7a833f49c68022c8a09e09c9ef" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 44587 } health_report { overall_health: UNKNOWN } } }
W20260925 22:36:34.415315 15154 tablet.cc:2369] T fef6b640e9704c899e5e0731e58c5306 P c99f2a7a833f49c68022c8a09e09c9ef: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:36:34.425485 15178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:34.422121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45986 (local address 127.6.246.194:36953)
0925 22:36:34.422242 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:34.422245 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:34.422259 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:36:34.422402 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:34.422405 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:34.422457 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:34.422553 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:36:34.422752 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:34.423244 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:34.424036 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:34.424243 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:34.424322 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:34.424651 (+ 329us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:36:34.425126 (+ 475us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:36:34.425216 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:34.425319 (+ 103us) server_negotiation.cc:299] Negotiation successful
0925 22:36:34.425371 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 22:36:34.430855 15107 raft_consensus.cc:1261] T fef6b640e9704c899e5e0731e58c5306 P c99f2a7a833f49c68022c8a09e09c9ef [term 1 FOLLOWER]: Refusing update from remote peer 063799ab6b434788be1447722615f67d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:36:34.430879 14827 raft_consensus.cc:1261] T fef6b640e9704c899e5e0731e58c5306 P e35ce5c2bbc9406e86e15892c07a97fd [term 1 FOLLOWER]: Refusing update from remote peer 063799ab6b434788be1447722615f67d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:36:34.431483 15189 consensus_queue.cc:1035] T fef6b640e9704c899e5e0731e58c5306 P 063799ab6b434788be1447722615f67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c99f2a7a833f49c68022c8a09e09c9ef" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 44587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:36:34.431699 15191 consensus_queue.cc:1035] T fef6b640e9704c899e5e0731e58c5306 P 063799ab6b434788be1447722615f67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e35ce5c2bbc9406e86e15892c07a97fd" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 38857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:36:34.442318 15196 mvcc.cc:205] Tried to move back new op lower bound from 7333379253982404608 to 7333379253774647296. Current Snapshot: MvccSnapshot[applied={T|T < 7333379253982404608 or (T in {7333379253982404608})}]
I20260925 22:36:34.447145 14670 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:36:34.447310 14670 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:36:34.449707 14670 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 302768cab249462291567a64833dd1a1: Sending DeleteTablet for 3 replicas of tablet fef6b640e9704c899e5e0731e58c5306
I20260925 22:36:34.450445 14947 tablet_service.cc:1514] Processing DeleteTablet for tablet fef6b640e9704c899e5e0731e58c5306 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:36:34 UTC) from {username='oryx', principal='oryx/127.6.246.254@KRBTEST.COM'} at 127.0.0.1:45972
I20260925 22:36:34.450635 14807 tablet_service.cc:1514] Processing DeleteTablet for tablet fef6b640e9704c899e5e0731e58c5306 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:36:34 UTC) from {username='oryx', principal='oryx/127.6.246.254@KRBTEST.COM'} at 127.0.0.1:58542
I20260925 22:36:34.451042 15205 tablet_replica.cc:330] T fef6b640e9704c899e5e0731e58c5306 P e35ce5c2bbc9406e86e15892c07a97fd: stopping tablet replica
I20260925 22:36:34.451061 15087 tablet_service.cc:1514] Processing DeleteTablet for tablet fef6b640e9704c899e5e0731e58c5306 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:36:34 UTC) from {username='oryx', principal='oryx/127.6.246.254@KRBTEST.COM'} at 127.0.0.1:54576
I20260925 22:36:34.451128 14670 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50324:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 22:36:34.451488 15206 tablet_replica.cc:330] T fef6b640e9704c899e5e0731e58c5306 P c99f2a7a833f49c68022c8a09e09c9ef: stopping tablet replica
W20260925 22:36:34.451526 14670 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:36:34.451759 15206 raft_consensus.cc:2229] T fef6b640e9704c899e5e0731e58c5306 P c99f2a7a833f49c68022c8a09e09c9ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:36:34.451910 15205 raft_consensus.cc:2229] T fef6b640e9704c899e5e0731e58c5306 P e35ce5c2bbc9406e86e15892c07a97fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:36:34.452061 15206 raft_consensus.cc:2258] T fef6b640e9704c899e5e0731e58c5306 P c99f2a7a833f49c68022c8a09e09c9ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:36:34.452301 15205 raft_consensus.cc:2258] T fef6b640e9704c899e5e0731e58c5306 P e35ce5c2bbc9406e86e15892c07a97fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:36:34.452633 15204 tablet_replica.cc:330] T fef6b640e9704c899e5e0731e58c5306 P 063799ab6b434788be1447722615f67d: stopping tablet replica
I20260925 22:36:34.452934 15204 raft_consensus.cc:2229] T fef6b640e9704c899e5e0731e58c5306 P 063799ab6b434788be1447722615f67d [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:36:34.453320 15204 raft_consensus.cc:2258] T fef6b640e9704c899e5e0731e58c5306 P 063799ab6b434788be1447722615f67d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:36:34.453890 15205 ts_tablet_manager.cc:1905] T fef6b640e9704c899e5e0731e58c5306 P e35ce5c2bbc9406e86e15892c07a97fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:36:34.454710 15204 ts_tablet_manager.cc:1905] T fef6b640e9704c899e5e0731e58c5306 P 063799ab6b434788be1447722615f67d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:36:34.457135 14807 tablet_service.cc:1467] Processing CreateTablet for tablet b2b2870876a5435eb57390f706dc7364 (DEFAULT_TABLE table=test-table [id=6e724673d52547d4a077daf0c58842fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:36:34.457410 14807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2b2870876a5435eb57390f706dc7364. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:36:34.458107 15205 ts_tablet_manager.cc:1918] T fef6b640e9704c899e5e0731e58c5306 P e35ce5c2bbc9406e86e15892c07a97fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:36:34.458196 15205 log.cc:1196] T fef6b640e9704c899e5e0731e58c5306 P e35ce5c2bbc9406e86e15892c07a97fd: Deleting WAL directory at /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/minicluster-data/ts-0/wal/wals/fef6b640e9704c899e5e0731e58c5306
I20260925 22:36:34.458505 15205 ts_tablet_manager.cc:1939] T fef6b640e9704c899e5e0731e58c5306 P e35ce5c2bbc9406e86e15892c07a97fd: Deleting consensus metadata
I20260925 22:36:34.459553 15184 tablet_bootstrap.cc:492] T b2b2870876a5435eb57390f706dc7364 P e35ce5c2bbc9406e86e15892c07a97fd: Bootstrap starting.
I20260925 22:36:34.459761 15087 tablet_service.cc:1467] Processing CreateTablet for tablet b2b2870876a5435eb57390f706dc7364 (DEFAULT_TABLE table=test-table [id=6e724673d52547d4a077daf0c58842fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:36:34.459998 15087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2b2870876a5435eb57390f706dc7364. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:36:34.460683 14947 tablet_service.cc:1467] Processing CreateTablet for tablet b2b2870876a5435eb57390f706dc7364 (DEFAULT_TABLE table=test-table [id=6e724673d52547d4a077daf0c58842fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:36:34.460753 14654 catalog_manager.cc:4841] TS e35ce5c2bbc9406e86e15892c07a97fd (127.6.246.193:38857): tablet fef6b640e9704c899e5e0731e58c5306 (table test-table [id=ca2f52055e514f10adc76fc6f08e88c2]) successfully deleted
I20260925 22:36:34.460904 14947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2b2870876a5435eb57390f706dc7364. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:36:34.462482 15181 tablet_bootstrap.cc:492] T b2b2870876a5435eb57390f706dc7364 P c99f2a7a833f49c68022c8a09e09c9ef: Bootstrap starting.
I20260925 22:36:34.462971 15182 tablet_bootstrap.cc:492] T b2b2870876a5435eb57390f706dc7364 P 063799ab6b434788be1447722615f67d: Bootstrap starting.
I20260925 22:36:34.463231 15184 tablet_bootstrap.cc:654] T b2b2870876a5435eb57390f706dc7364 P e35ce5c2bbc9406e86e15892c07a97fd: Neither blocks nor log segments found. Creating new log.
I20260925 22:36:34.463433 15181 tablet_bootstrap.cc:654] T b2b2870876a5435eb57390f706dc7364 P c99f2a7a833f49c68022c8a09e09c9ef: Neither blocks nor log segments found. Creating new log.
I20260925 22:36:34.463901 15182 tablet_bootstrap.cc:654] T b2b2870876a5435eb57390f706dc7364 P 063799ab6b434788be1447722615f67d: Neither blocks nor log segments found. Creating new log.
I20260925 22:36:34.464583 15184 tablet_bootstrap.cc:492] T b2b2870876a5435eb57390f706dc7364 P e35ce5c2bbc9406e86e15892c07a97fd: No bootstrap required, opened a new log
I20260925 22:36:34.464687 15184 ts_tablet_manager.cc:1397] T b2b2870876a5435eb57390f706dc7364 P e35ce5c2bbc9406e86e15892c07a97fd: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260925 22:36:34.465174 15184 raft_consensus.cc:348] T b2b2870876a5435eb57390f706dc7364 P e35ce5c2bbc9406e86e15892c07a97fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35ce5c2bbc9406e86e15892c07a97fd" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 38857 } } peers { permanent_uuid: "063799ab6b434788be1447722615f67d" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 36953 } } peers { permanent_uuid: "c99f2a7a833f49c68022c8a09e09c9ef" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 44587 } }
I20260925 22:36:34.465281 15184 raft_consensus.cc:374] T b2b2870876a5435eb57390f706dc7364 P e35ce5c2bbc9406e86e15892c07a97fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:36:34.465318 15184 raft_consensus.cc:729] T b2b2870876a5435eb57390f706dc7364 P e35ce5c2bbc9406e86e15892c07a97fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e35ce5c2bbc9406e86e15892c07a97fd, State: Initialized, Role: FOLLOWER
I20260925 22:36:34.465456 15184 consensus_queue.cc:260] T b2b2870876a5435eb57390f706dc7364 P e35ce5c2bbc9406e86e15892c07a97fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35ce5c2bbc9406e86e15892c07a97fd" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 38857 } } peers { permanent_uuid: "063799ab6b434788be1447722615f67d" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 36953 } } peers { permanent_uuid: "c99f2a7a833f49c68022c8a09e09c9ef" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 44587 } }
I20260925 22:36:34.465737 15184 ts_tablet_manager.cc:1428] T b2b2870876a5435eb57390f706dc7364 P e35ce5c2bbc9406e86e15892c07a97fd: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260925 22:36:34.465978 15206 ts_tablet_manager.cc:1905] T fef6b640e9704c899e5e0731e58c5306 P c99f2a7a833f49c68022c8a09e09c9ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:36:34.466914 15181 tablet_bootstrap.cc:492] T b2b2870876a5435eb57390f706dc7364 P c99f2a7a833f49c68022c8a09e09c9ef: No bootstrap required, opened a new log
I20260925 22:36:34.466997 15181 ts_tablet_manager.cc:1397] T b2b2870876a5435eb57390f706dc7364 P c99f2a7a833f49c68022c8a09e09c9ef: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260925 22:36:34.467465 15181 raft_consensus.cc:348] T b2b2870876a5435eb57390f706dc7364 P c99f2a7a833f49c68022c8a09e09c9ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35ce5c2bbc9406e86e15892c07a97fd" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 38857 } } peers { permanent_uuid: "063799ab6b434788be1447722615f67d" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 36953 } } peers { permanent_uuid: "c99f2a7a833f49c68022c8a09e09c9ef" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 44587 } }
I20260925 22:36:34.467566 15181 raft_consensus.cc:374] T b2b2870876a5435eb57390f706dc7364 P c99f2a7a833f49c68022c8a09e09c9ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:36:34.467604 15181 raft_consensus.cc:729] T b2b2870876a5435eb57390f706dc7364 P c99f2a7a833f49c68022c8a09e09c9ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c99f2a7a833f49c68022c8a09e09c9ef, State: Initialized, Role: FOLLOWER
I20260925 22:36:34.467718 15181 consensus_queue.cc:260] T b2b2870876a5435eb57390f706dc7364 P c99f2a7a833f49c68022c8a09e09c9ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35ce5c2bbc9406e86e15892c07a97fd" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 38857 } } peers { permanent_uuid: "063799ab6b434788be1447722615f67d" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 36953 } } peers { permanent_uuid: "c99f2a7a833f49c68022c8a09e09c9ef" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 44587 } }
I20260925 22:36:34.467976 15181 ts_tablet_manager.cc:1428] T b2b2870876a5435eb57390f706dc7364 P c99f2a7a833f49c68022c8a09e09c9ef: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:36:34.468523 15206 ts_tablet_manager.cc:1918] T fef6b640e9704c899e5e0731e58c5306 P c99f2a7a833f49c68022c8a09e09c9ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:36:34.468672 15182 tablet_bootstrap.cc:492] T b2b2870876a5435eb57390f706dc7364 P 063799ab6b434788be1447722615f67d: No bootstrap required, opened a new log
I20260925 22:36:34.468756 15182 ts_tablet_manager.cc:1397] T b2b2870876a5435eb57390f706dc7364 P 063799ab6b434788be1447722615f67d: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260925 22:36:34.468837 15206 log.cc:1196] T fef6b640e9704c899e5e0731e58c5306 P c99f2a7a833f49c68022c8a09e09c9ef: Deleting WAL directory at /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/minicluster-data/ts-2/wal/wals/fef6b640e9704c899e5e0731e58c5306
I20260925 22:36:34.469154 15206 ts_tablet_manager.cc:1939] T fef6b640e9704c899e5e0731e58c5306 P c99f2a7a833f49c68022c8a09e09c9ef: Deleting consensus metadata
I20260925 22:36:34.469259 15182 raft_consensus.cc:348] T b2b2870876a5435eb57390f706dc7364 P 063799ab6b434788be1447722615f67d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35ce5c2bbc9406e86e15892c07a97fd" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 38857 } } peers { permanent_uuid: "063799ab6b434788be1447722615f67d" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 36953 } } peers { permanent_uuid: "c99f2a7a833f49c68022c8a09e09c9ef" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 44587 } }
I20260925 22:36:34.469372 15182 raft_consensus.cc:374] T b2b2870876a5435eb57390f706dc7364 P 063799ab6b434788be1447722615f67d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:36:34.469406 15182 raft_consensus.cc:729] T b2b2870876a5435eb57390f706dc7364 P 063799ab6b434788be1447722615f67d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 063799ab6b434788be1447722615f67d, State: Initialized, Role: FOLLOWER
I20260925 22:36:34.469517 15182 consensus_queue.cc:260] T b2b2870876a5435eb57390f706dc7364 P 063799ab6b434788be1447722615f67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35ce5c2bbc9406e86e15892c07a97fd" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 38857 } } peers { permanent_uuid: "063799ab6b434788be1447722615f67d" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 36953 } } peers { permanent_uuid: "c99f2a7a833f49c68022c8a09e09c9ef" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 44587 } }
I20260925 22:36:34.469736 15182 ts_tablet_manager.cc:1428] T b2b2870876a5435eb57390f706dc7364 P 063799ab6b434788be1447722615f67d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:36:34.470363 14654 catalog_manager.cc:4841] TS c99f2a7a833f49c68022c8a09e09c9ef (127.6.246.195:44587): tablet fef6b640e9704c899e5e0731e58c5306 (table test-table [id=ca2f52055e514f10adc76fc6f08e88c2]) successfully deleted
I20260925 22:36:34.473780 15204 ts_tablet_manager.cc:1918] T fef6b640e9704c899e5e0731e58c5306 P 063799ab6b434788be1447722615f67d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:36:34.473886 15204 log.cc:1196] T fef6b640e9704c899e5e0731e58c5306 P 063799ab6b434788be1447722615f67d: Deleting WAL directory at /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/minicluster-data/ts-1/wal/wals/fef6b640e9704c899e5e0731e58c5306
I20260925 22:36:34.474162 15204 ts_tablet_manager.cc:1939] T fef6b640e9704c899e5e0731e58c5306 P 063799ab6b434788be1447722615f67d: Deleting consensus metadata
I20260925 22:36:34.475057 14655 catalog_manager.cc:4841] TS 063799ab6b434788be1447722615f67d (127.6.246.194:36953): tablet fef6b640e9704c899e5e0731e58c5306 (table test-table [id=ca2f52055e514f10adc76fc6f08e88c2]) successfully deleted
W20260925 22:36:34.486209 14874 tablet.cc:2369] T b2b2870876a5435eb57390f706dc7364 P e35ce5c2bbc9406e86e15892c07a97fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:36:34.571519 15199 raft_consensus.cc:482] T b2b2870876a5435eb57390f706dc7364 P 063799ab6b434788be1447722615f67d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:36:34.571695 15199 raft_consensus.cc:504] T b2b2870876a5435eb57390f706dc7364 P 063799ab6b434788be1447722615f67d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35ce5c2bbc9406e86e15892c07a97fd" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 38857 } } peers { permanent_uuid: "063799ab6b434788be1447722615f67d" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 36953 } } peers { permanent_uuid: "c99f2a7a833f49c68022c8a09e09c9ef" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 44587 } }
I20260925 22:36:34.572171 15199 leader_election.cc:288] T b2b2870876a5435eb57390f706dc7364 P 063799ab6b434788be1447722615f67d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e35ce5c2bbc9406e86e15892c07a97fd (127.6.246.193:38857), c99f2a7a833f49c68022c8a09e09c9ef (127.6.246.195:44587)
I20260925 22:36:34.572750 15107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2b2870876a5435eb57390f706dc7364" candidate_uuid: "063799ab6b434788be1447722615f67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c99f2a7a833f49c68022c8a09e09c9ef" is_pre_election: true
I20260925 22:36:34.572799 14827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2b2870876a5435eb57390f706dc7364" candidate_uuid: "063799ab6b434788be1447722615f67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e35ce5c2bbc9406e86e15892c07a97fd" is_pre_election: true
I20260925 22:36:34.572916 15107 raft_consensus.cc:2454] T b2b2870876a5435eb57390f706dc7364 P c99f2a7a833f49c68022c8a09e09c9ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 063799ab6b434788be1447722615f67d in term 0.
I20260925 22:36:34.572984 14827 raft_consensus.cc:2454] T b2b2870876a5435eb57390f706dc7364 P e35ce5c2bbc9406e86e15892c07a97fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 063799ab6b434788be1447722615f67d in term 0.
I20260925 22:36:34.573276 14898 leader_election.cc:302] T b2b2870876a5435eb57390f706dc7364 P 063799ab6b434788be1447722615f67d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 063799ab6b434788be1447722615f67d, c99f2a7a833f49c68022c8a09e09c9ef; no voters:
I20260925 22:36:34.573505 15199 raft_consensus.cc:2781] T b2b2870876a5435eb57390f706dc7364 P 063799ab6b434788be1447722615f67d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:36:34.573577 15199 raft_consensus.cc:482] T b2b2870876a5435eb57390f706dc7364 P 063799ab6b434788be1447722615f67d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:36:34.573632 15199 raft_consensus.cc:3037] T b2b2870876a5435eb57390f706dc7364 P 063799ab6b434788be1447722615f67d [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:36:34.574674 15199 raft_consensus.cc:504] T b2b2870876a5435eb57390f706dc7364 P 063799ab6b434788be1447722615f67d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35ce5c2bbc9406e86e15892c07a97fd" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 38857 } } peers { permanent_uuid: "063799ab6b434788be1447722615f67d" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 36953 } } peers { permanent_uuid: "c99f2a7a833f49c68022c8a09e09c9ef" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 44587 } }
I20260925 22:36:34.575012 15199 leader_election.cc:288] T b2b2870876a5435eb57390f706dc7364 P 063799ab6b434788be1447722615f67d [CANDIDATE]: Term 1 election: Requested vote from peers e35ce5c2bbc9406e86e15892c07a97fd (127.6.246.193:38857), c99f2a7a833f49c68022c8a09e09c9ef (127.6.246.195:44587)
I20260925 22:36:34.575503 15107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2b2870876a5435eb57390f706dc7364" candidate_uuid: "063799ab6b434788be1447722615f67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c99f2a7a833f49c68022c8a09e09c9ef"
I20260925 22:36:34.575502 14827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2b2870876a5435eb57390f706dc7364" candidate_uuid: "063799ab6b434788be1447722615f67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e35ce5c2bbc9406e86e15892c07a97fd"
I20260925 22:36:34.575623 15107 raft_consensus.cc:3037] T b2b2870876a5435eb57390f706dc7364 P c99f2a7a833f49c68022c8a09e09c9ef [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:36:34.575623 14827 raft_consensus.cc:3037] T b2b2870876a5435eb57390f706dc7364 P e35ce5c2bbc9406e86e15892c07a97fd [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:36:34.576828 14827 raft_consensus.cc:2454] T b2b2870876a5435eb57390f706dc7364 P e35ce5c2bbc9406e86e15892c07a97fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 063799ab6b434788be1447722615f67d in term 1.
I20260925 22:36:34.576884 15107 raft_consensus.cc:2454] T b2b2870876a5435eb57390f706dc7364 P c99f2a7a833f49c68022c8a09e09c9ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 063799ab6b434788be1447722615f67d in term 1.
I20260925 22:36:34.577140 14898 leader_election.cc:302] T b2b2870876a5435eb57390f706dc7364 P 063799ab6b434788be1447722615f67d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 063799ab6b434788be1447722615f67d, c99f2a7a833f49c68022c8a09e09c9ef; no voters:
I20260925 22:36:34.577306 15199 raft_consensus.cc:2781] T b2b2870876a5435eb57390f706dc7364 P 063799ab6b434788be1447722615f67d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:36:34.577414 15199 raft_consensus.cc:686] T b2b2870876a5435eb57390f706dc7364 P 063799ab6b434788be1447722615f67d [term 1 LEADER]: Becoming Leader. State: Replica: 063799ab6b434788be1447722615f67d, State: Running, Role: LEADER
I20260925 22:36:34.577553 15199 consensus_queue.cc:237] T b2b2870876a5435eb57390f706dc7364 P 063799ab6b434788be1447722615f67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35ce5c2bbc9406e86e15892c07a97fd" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 38857 } } peers { permanent_uuid: "063799ab6b434788be1447722615f67d" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 36953 } } peers { permanent_uuid: "c99f2a7a833f49c68022c8a09e09c9ef" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 44587 } }
I20260925 22:36:34.579447 14670 catalog_manager.cc:5495] T b2b2870876a5435eb57390f706dc7364 P 063799ab6b434788be1447722615f67d reported cstate change: term changed from 0 to 1, leader changed from <none> to 063799ab6b434788be1447722615f67d (127.6.246.194). New cstate: current_term: 1 leader_uuid: "063799ab6b434788be1447722615f67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35ce5c2bbc9406e86e15892c07a97fd" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 38857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "063799ab6b434788be1447722615f67d" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 36953 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c99f2a7a833f49c68022c8a09e09c9ef" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 44587 } health_report { overall_health: UNKNOWN } } }
Sep 25 22:36:34 dist-test-slave-wjdr krb5kdc[14625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375793, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.246.254@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM
I20260925 22:36:34.642380 15216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:34.634234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:38821 (local address 127.0.0.1:50336)
0925 22:36:34.634518 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:34.634531 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:34.634631 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:34.634840 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:34.634842 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:34.635205 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:34.635296 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:34.635301 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:34.636012 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:34.636015 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:34.636641 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:34.636650 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:34.637032 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:34.637035 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:34.637098 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:34.637606 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:34.637617 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:34.638978 (+ 1361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:36:34.640878 (+ 1900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:34.640886 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:34.640891 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:34.641144 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:34.641434 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:34.641436 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:34.641438 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:34.641519 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:34.641853 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:36:34.641859 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:36:34.641962 (+ 103us) client_negotiation.cc:769] Sending connection context
0925 22:36:34.642056 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 22:36:34.642180 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":123,"threads_started":1}
I20260925 22:36:34.642380 14719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:34.634269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50336 (local address 127.6.246.254:38821)
0925 22:36:34.634414 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:34.634419 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:34.634603 (+ 184us) server_negotiation.cc:407] Connection header received
0925 22:36:34.634698 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:34.634701 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:34.634740 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:34.634807 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:34.635401 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:34.635913 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:34.636736 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:34.636938 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:34.637011 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:34.639110 (+ 2099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:34.639114 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:34.639116 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:34.639174 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:34.640774 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:34.641265 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:34.641267 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:34.641269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:34.641335 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:34.641598 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:34.641600 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:34.641601 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:34.641764 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:34.641835 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:34.642109 (+ 274us) server_negotiation.cc:299] Negotiation successful
0925 22:36:34.642234 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":52}
I20260925 22:36:34.646230 14670 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.6.246.254@KRBTEST.COM'} at 127.0.0.1:50336:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 22:36:34.646962 14670 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:36:34.651955 14947 tablet_service.cc:1467] Processing CreateTablet for tablet 057ccfc562754c5fa2b789c8056e05bf (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f2a1ad2b2b7e40bd87b9d73fe20bb685]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 22:36:34.652001 14807 tablet_service.cc:1467] Processing CreateTablet for tablet 057ccfc562754c5fa2b789c8056e05bf (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f2a1ad2b2b7e40bd87b9d73fe20bb685]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 22:36:34.652107 15087 tablet_service.cc:1467] Processing CreateTablet for tablet 057ccfc562754c5fa2b789c8056e05bf (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f2a1ad2b2b7e40bd87b9d73fe20bb685]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 22:36:34.652352 14947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 057ccfc562754c5fa2b789c8056e05bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:36:34.652375 14807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 057ccfc562754c5fa2b789c8056e05bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:36:34.652429 15087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 057ccfc562754c5fa2b789c8056e05bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:36:34.654807 15182 tablet_bootstrap.cc:492] T 057ccfc562754c5fa2b789c8056e05bf P 063799ab6b434788be1447722615f67d: Bootstrap starting.
I20260925 22:36:34.655185 15181 tablet_bootstrap.cc:492] T 057ccfc562754c5fa2b789c8056e05bf P c99f2a7a833f49c68022c8a09e09c9ef: Bootstrap starting.
I20260925 22:36:34.655719 15182 tablet_bootstrap.cc:654] T 057ccfc562754c5fa2b789c8056e05bf P 063799ab6b434788be1447722615f67d: Neither blocks nor log segments found. Creating new log.
I20260925 22:36:34.656150 15181 tablet_bootstrap.cc:654] T 057ccfc562754c5fa2b789c8056e05bf P c99f2a7a833f49c68022c8a09e09c9ef: Neither blocks nor log segments found. Creating new log.
I20260925 22:36:34.656591 15184 tablet_bootstrap.cc:492] T 057ccfc562754c5fa2b789c8056e05bf P e35ce5c2bbc9406e86e15892c07a97fd: Bootstrap starting.
I20260925 22:36:34.656883 15182 tablet_bootstrap.cc:492] T 057ccfc562754c5fa2b789c8056e05bf P 063799ab6b434788be1447722615f67d: No bootstrap required, opened a new log
I20260925 22:36:34.656967 15182 ts_tablet_manager.cc:1397] T 057ccfc562754c5fa2b789c8056e05bf P 063799ab6b434788be1447722615f67d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 22:36:34.657354 15181 tablet_bootstrap.cc:492] T 057ccfc562754c5fa2b789c8056e05bf P c99f2a7a833f49c68022c8a09e09c9ef: No bootstrap required, opened a new log
I20260925 22:36:34.657343 15182 raft_consensus.cc:348] T 057ccfc562754c5fa2b789c8056e05bf P 063799ab6b434788be1447722615f67d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35ce5c2bbc9406e86e15892c07a97fd" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 38857 } } peers { permanent_uuid: "063799ab6b434788be1447722615f67d" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 36953 } } peers { permanent_uuid: "c99f2a7a833f49c68022c8a09e09c9ef" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 44587 } }
I20260925 22:36:34.657425 15182 raft_consensus.cc:374] T 057ccfc562754c5fa2b789c8056e05bf P 063799ab6b434788be1447722615f67d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:36:34.657431 15181 ts_tablet_manager.cc:1397] T 057ccfc562754c5fa2b789c8056e05bf P c99f2a7a833f49c68022c8a09e09c9ef: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260925 22:36:34.657446 15182 raft_consensus.cc:729] T 057ccfc562754c5fa2b789c8056e05bf P 063799ab6b434788be1447722615f67d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 063799ab6b434788be1447722615f67d, State: Initialized, Role: FOLLOWER
I20260925 22:36:34.657512 15184 tablet_bootstrap.cc:654] T 057ccfc562754c5fa2b789c8056e05bf P e35ce5c2bbc9406e86e15892c07a97fd: Neither blocks nor log segments found. Creating new log.
I20260925 22:36:34.657570 15182 consensus_queue.cc:260] T 057ccfc562754c5fa2b789c8056e05bf P 063799ab6b434788be1447722615f67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35ce5c2bbc9406e86e15892c07a97fd" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 38857 } } peers { permanent_uuid: "063799ab6b434788be1447722615f67d" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 36953 } } peers { permanent_uuid: "c99f2a7a833f49c68022c8a09e09c9ef" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 44587 } }
I20260925 22:36:34.657835 15182 ts_tablet_manager.cc:1428] T 057ccfc562754c5fa2b789c8056e05bf P 063799ab6b434788be1447722615f67d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260925 22:36:34.657778 15191 tablet_replica.cc:439] T 057ccfc562754c5fa2b789c8056e05bf P 063799ab6b434788be1447722615f67d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35ce5c2bbc9406e86e15892c07a97fd" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 38857 } } peers { permanent_uuid: "063799ab6b434788be1447722615f67d" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 36953 } } peers { permanent_uuid: "c99f2a7a833f49c68022c8a09e09c9ef" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 44587 } } }
I20260925 22:36:34.657900 15191 tablet_replica.cc:442] T 057ccfc562754c5fa2b789c8056e05bf P 063799ab6b434788be1447722615f67d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:36:34.657872 15181 raft_consensus.cc:348] T 057ccfc562754c5fa2b789c8056e05bf P c99f2a7a833f49c68022c8a09e09c9ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35ce5c2bbc9406e86e15892c07a97fd" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 38857 } } peers { permanent_uuid: "063799ab6b434788be1447722615f67d" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 36953 } } peers { permanent_uuid: "c99f2a7a833f49c68022c8a09e09c9ef" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 44587 } }
I20260925 22:36:34.657987 15181 raft_consensus.cc:374] T 057ccfc562754c5fa2b789c8056e05bf P c99f2a7a833f49c68022c8a09e09c9ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:36:34.658030 15181 raft_consensus.cc:729] T 057ccfc562754c5fa2b789c8056e05bf P c99f2a7a833f49c68022c8a09e09c9ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c99f2a7a833f49c68022c8a09e09c9ef, State: Initialized, Role: FOLLOWER
I20260925 22:36:34.658175 15181 consensus_queue.cc:260] T 057ccfc562754c5fa2b789c8056e05bf P c99f2a7a833f49c68022c8a09e09c9ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35ce5c2bbc9406e86e15892c07a97fd" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 38857 } } peers { permanent_uuid: "063799ab6b434788be1447722615f67d" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 36953 } } peers { permanent_uuid: "c99f2a7a833f49c68022c8a09e09c9ef" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 44587 } }
I20260925 22:36:34.658449 15181 ts_tablet_manager.cc:1428] T 057ccfc562754c5fa2b789c8056e05bf P c99f2a7a833f49c68022c8a09e09c9ef: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 22:36:34.658402 15187 tablet_replica.cc:439] T 057ccfc562754c5fa2b789c8056e05bf P c99f2a7a833f49c68022c8a09e09c9ef: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35ce5c2bbc9406e86e15892c07a97fd" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 38857 } } peers { permanent_uuid: "063799ab6b434788be1447722615f67d" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 36953 } } peers { permanent_uuid: "c99f2a7a833f49c68022c8a09e09c9ef" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 44587 } } }
I20260925 22:36:34.658532 15187 tablet_replica.cc:442] T 057ccfc562754c5fa2b789c8056e05bf P c99f2a7a833f49c68022c8a09e09c9ef: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:36:34.658982 15184 tablet_bootstrap.cc:492] T 057ccfc562754c5fa2b789c8056e05bf P e35ce5c2bbc9406e86e15892c07a97fd: No bootstrap required, opened a new log
I20260925 22:36:34.659106 15184 ts_tablet_manager.cc:1397] T 057ccfc562754c5fa2b789c8056e05bf P e35ce5c2bbc9406e86e15892c07a97fd: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260925 22:36:34.659556 15184 raft_consensus.cc:348] T 057ccfc562754c5fa2b789c8056e05bf P e35ce5c2bbc9406e86e15892c07a97fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35ce5c2bbc9406e86e15892c07a97fd" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 38857 } } peers { permanent_uuid: "063799ab6b434788be1447722615f67d" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 36953 } } peers { permanent_uuid: "c99f2a7a833f49c68022c8a09e09c9ef" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 44587 } }
I20260925 22:36:34.659699 15184 raft_consensus.cc:374] T 057ccfc562754c5fa2b789c8056e05bf P e35ce5c2bbc9406e86e15892c07a97fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:36:34.659760 15184 raft_consensus.cc:729] T 057ccfc562754c5fa2b789c8056e05bf P e35ce5c2bbc9406e86e15892c07a97fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e35ce5c2bbc9406e86e15892c07a97fd, State: Initialized, Role: FOLLOWER
I20260925 22:36:34.659895 15184 consensus_queue.cc:260] T 057ccfc562754c5fa2b789c8056e05bf P e35ce5c2bbc9406e86e15892c07a97fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35ce5c2bbc9406e86e15892c07a97fd" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 38857 } } peers { permanent_uuid: "063799ab6b434788be1447722615f67d" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 36953 } } peers { permanent_uuid: "c99f2a7a833f49c68022c8a09e09c9ef" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 44587 } }
I20260925 22:36:34.660149 15184 ts_tablet_manager.cc:1428] T 057ccfc562754c5fa2b789c8056e05bf P e35ce5c2bbc9406e86e15892c07a97fd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 22:36:34.660137 15188 tablet_replica.cc:439] T 057ccfc562754c5fa2b789c8056e05bf P e35ce5c2bbc9406e86e15892c07a97fd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35ce5c2bbc9406e86e15892c07a97fd" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 38857 } } peers { permanent_uuid: "063799ab6b434788be1447722615f67d" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 36953 } } peers { permanent_uuid: "c99f2a7a833f49c68022c8a09e09c9ef" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 44587 } } }
I20260925 22:36:34.660277 15188 tablet_replica.cc:442] T 057ccfc562754c5fa2b789c8056e05bf P e35ce5c2bbc9406e86e15892c07a97fd: This TxnStatusTablet replica's current role is: FOLLOWER
W20260925 22:36:34.695298 15014 tablet.cc:2369] T 057ccfc562754c5fa2b789c8056e05bf P 063799ab6b434788be1447722615f67d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:36:34.747066 15191 raft_consensus.cc:482] T 057ccfc562754c5fa2b789c8056e05bf P 063799ab6b434788be1447722615f67d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:36:34.747220 15191 raft_consensus.cc:504] T 057ccfc562754c5fa2b789c8056e05bf P 063799ab6b434788be1447722615f67d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35ce5c2bbc9406e86e15892c07a97fd" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 38857 } } peers { permanent_uuid: "063799ab6b434788be1447722615f67d" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 36953 } } peers { permanent_uuid: "c99f2a7a833f49c68022c8a09e09c9ef" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 44587 } }
I20260925 22:36:34.747702 15191 leader_election.cc:288] T 057ccfc562754c5fa2b789c8056e05bf P 063799ab6b434788be1447722615f67d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e35ce5c2bbc9406e86e15892c07a97fd (127.6.246.193:38857), c99f2a7a833f49c68022c8a09e09c9ef (127.6.246.195:44587)
I20260925 22:36:34.748147 14827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "057ccfc562754c5fa2b789c8056e05bf" candidate_uuid: "063799ab6b434788be1447722615f67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e35ce5c2bbc9406e86e15892c07a97fd" is_pre_election: true
I20260925 22:36:34.748220 15107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "057ccfc562754c5fa2b789c8056e05bf" candidate_uuid: "063799ab6b434788be1447722615f67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c99f2a7a833f49c68022c8a09e09c9ef" is_pre_election: true
I20260925 22:36:34.748293 14827 raft_consensus.cc:2454] T 057ccfc562754c5fa2b789c8056e05bf P e35ce5c2bbc9406e86e15892c07a97fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 063799ab6b434788be1447722615f67d in term 0.
I20260925 22:36:34.748353 15107 raft_consensus.cc:2454] T 057ccfc562754c5fa2b789c8056e05bf P c99f2a7a833f49c68022c8a09e09c9ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 063799ab6b434788be1447722615f67d in term 0.
I20260925 22:36:34.748668 14898 leader_election.cc:302] T 057ccfc562754c5fa2b789c8056e05bf P 063799ab6b434788be1447722615f67d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 063799ab6b434788be1447722615f67d, c99f2a7a833f49c68022c8a09e09c9ef; no voters:
I20260925 22:36:34.748899 15191 raft_consensus.cc:2781] T 057ccfc562754c5fa2b789c8056e05bf P 063799ab6b434788be1447722615f67d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:36:34.748967 15191 raft_consensus.cc:482] T 057ccfc562754c5fa2b789c8056e05bf P 063799ab6b434788be1447722615f67d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:36:34.749007 15191 raft_consensus.cc:3037] T 057ccfc562754c5fa2b789c8056e05bf P 063799ab6b434788be1447722615f67d [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:36:34.750018 15191 raft_consensus.cc:504] T 057ccfc562754c5fa2b789c8056e05bf P 063799ab6b434788be1447722615f67d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35ce5c2bbc9406e86e15892c07a97fd" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 38857 } } peers { permanent_uuid: "063799ab6b434788be1447722615f67d" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 36953 } } peers { permanent_uuid: "c99f2a7a833f49c68022c8a09e09c9ef" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 44587 } }
I20260925 22:36:34.750345 15191 leader_election.cc:288] T 057ccfc562754c5fa2b789c8056e05bf P 063799ab6b434788be1447722615f67d [CANDIDATE]: Term 1 election: Requested vote from peers e35ce5c2bbc9406e86e15892c07a97fd (127.6.246.193:38857), c99f2a7a833f49c68022c8a09e09c9ef (127.6.246.195:44587)
I20260925 22:36:34.750723 14827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "057ccfc562754c5fa2b789c8056e05bf" candidate_uuid: "063799ab6b434788be1447722615f67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e35ce5c2bbc9406e86e15892c07a97fd"
I20260925 22:36:34.750728 15107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "057ccfc562754c5fa2b789c8056e05bf" candidate_uuid: "063799ab6b434788be1447722615f67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c99f2a7a833f49c68022c8a09e09c9ef"
I20260925 22:36:34.750833 14827 raft_consensus.cc:3037] T 057ccfc562754c5fa2b789c8056e05bf P e35ce5c2bbc9406e86e15892c07a97fd [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:36:34.750941 15107 raft_consensus.cc:3037] T 057ccfc562754c5fa2b789c8056e05bf P c99f2a7a833f49c68022c8a09e09c9ef [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:36:34.751713 14827 raft_consensus.cc:2454] T 057ccfc562754c5fa2b789c8056e05bf P e35ce5c2bbc9406e86e15892c07a97fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 063799ab6b434788be1447722615f67d in term 1.
I20260925 22:36:34.751889 15107 raft_consensus.cc:2454] T 057ccfc562754c5fa2b789c8056e05bf P c99f2a7a833f49c68022c8a09e09c9ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 063799ab6b434788be1447722615f67d in term 1.
I20260925 22:36:34.751998 14898 leader_election.cc:302] T 057ccfc562754c5fa2b789c8056e05bf P 063799ab6b434788be1447722615f67d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 063799ab6b434788be1447722615f67d, e35ce5c2bbc9406e86e15892c07a97fd; no voters:
I20260925 22:36:34.752170 15191 raft_consensus.cc:2781] T 057ccfc562754c5fa2b789c8056e05bf P 063799ab6b434788be1447722615f67d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:36:34.752301 15191 raft_consensus.cc:686] T 057ccfc562754c5fa2b789c8056e05bf P 063799ab6b434788be1447722615f67d [term 1 LEADER]: Becoming Leader. State: Replica: 063799ab6b434788be1447722615f67d, State: Running, Role: LEADER
I20260925 22:36:34.752480 15191 consensus_queue.cc:237] T 057ccfc562754c5fa2b789c8056e05bf P 063799ab6b434788be1447722615f67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35ce5c2bbc9406e86e15892c07a97fd" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 38857 } } peers { permanent_uuid: "063799ab6b434788be1447722615f67d" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 36953 } } peers { permanent_uuid: "c99f2a7a833f49c68022c8a09e09c9ef" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 44587 } }
I20260925 22:36:34.753181 15199 tablet_replica.cc:439] T 057ccfc562754c5fa2b789c8056e05bf P 063799ab6b434788be1447722615f67d: TxnStatusTablet state changed. Reason: New leader 063799ab6b434788be1447722615f67d. Latest consensus state: current_term: 1 leader_uuid: "063799ab6b434788be1447722615f67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35ce5c2bbc9406e86e15892c07a97fd" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 38857 } } peers { permanent_uuid: "063799ab6b434788be1447722615f67d" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 36953 } } peers { permanent_uuid: "c99f2a7a833f49c68022c8a09e09c9ef" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 44587 } } }
I20260925 22:36:34.753306 15199 tablet_replica.cc:442] T 057ccfc562754c5fa2b789c8056e05bf P 063799ab6b434788be1447722615f67d: This TxnStatusTablet replica's current role is: LEADER
I20260925 22:36:34.753526 15221 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 22:36:34.753693 15221 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 22:36:34.754515 14670 catalog_manager.cc:5495] T 057ccfc562754c5fa2b789c8056e05bf P 063799ab6b434788be1447722615f67d reported cstate change: term changed from 0 to 1, leader changed from <none> to 063799ab6b434788be1447722615f67d (127.6.246.194). New cstate: current_term: 1 leader_uuid: "063799ab6b434788be1447722615f67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35ce5c2bbc9406e86e15892c07a97fd" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 38857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "063799ab6b434788be1447722615f67d" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 36953 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c99f2a7a833f49c68022c8a09e09c9ef" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 44587 } health_report { overall_health: UNKNOWN } } }
I20260925 22:36:34.786661 15216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:34.783248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.194:36953 (local address 127.0.0.1:45998)
0925 22:36:34.783398 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:34.783416 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:34.783517 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:34.783762 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:34.783764 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:34.783768 (+ 4us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 22:36:34.783862 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:34.783866 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:34.784868 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:34.784871 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:34.785467 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:34.785474 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:34.786039 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:34.786042 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:34.786107 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:34.786148 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 22:36:34.786430 (+ 282us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 22:36:34.786440 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 22:36:34.786491 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 22:36:34.786554 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 22:36:34.786682 15178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:34.783370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45998 (local address 127.6.246.194:36953)
0925 22:36:34.783488 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:34.783493 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:34.783508 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:36:34.783609 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:34.783612 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:34.783665 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:34.783756 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:36:34.783944 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:34.784750 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:34.785581 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:34.785931 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:34.786035 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:34.786271 (+ 236us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:36:34.786350 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:36:34.786421 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:34.786501 (+ 80us) server_negotiation.cc:299] Negotiation successful
0925 22:36:34.786549 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 22:36:34.789381 15107 raft_consensus.cc:1261] T 057ccfc562754c5fa2b789c8056e05bf P c99f2a7a833f49c68022c8a09e09c9ef [term 1 FOLLOWER]: Refusing update from remote peer 063799ab6b434788be1447722615f67d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:36:34.789446 14827 raft_consensus.cc:1261] T 057ccfc562754c5fa2b789c8056e05bf P e35ce5c2bbc9406e86e15892c07a97fd [term 1 FOLLOWER]: Refusing update from remote peer 063799ab6b434788be1447722615f67d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:36:34.789840 15199 consensus_queue.cc:1035] T 057ccfc562754c5fa2b789c8056e05bf P 063799ab6b434788be1447722615f67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c99f2a7a833f49c68022c8a09e09c9ef" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 44587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:36:34.790119 15199 consensus_queue.cc:1035] T 057ccfc562754c5fa2b789c8056e05bf P 063799ab6b434788be1447722615f67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e35ce5c2bbc9406e86e15892c07a97fd" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 38857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:36:34.791152 15187 tablet_replica.cc:439] T 057ccfc562754c5fa2b789c8056e05bf P c99f2a7a833f49c68022c8a09e09c9ef: TxnStatusTablet state changed. Reason: New leader 063799ab6b434788be1447722615f67d. Latest consensus state: current_term: 1 leader_uuid: "063799ab6b434788be1447722615f67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35ce5c2bbc9406e86e15892c07a97fd" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 38857 } } peers { permanent_uuid: "063799ab6b434788be1447722615f67d" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 36953 } } peers { permanent_uuid: "c99f2a7a833f49c68022c8a09e09c9ef" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 44587 } } }
I20260925 22:36:34.791258 15187 tablet_replica.cc:442] T 057ccfc562754c5fa2b789c8056e05bf P c99f2a7a833f49c68022c8a09e09c9ef: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:36:34.791348 15188 tablet_replica.cc:439] T 057ccfc562754c5fa2b789c8056e05bf P e35ce5c2bbc9406e86e15892c07a97fd: TxnStatusTablet state changed. Reason: New leader 063799ab6b434788be1447722615f67d. Latest consensus state: current_term: 1 leader_uuid: "063799ab6b434788be1447722615f67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35ce5c2bbc9406e86e15892c07a97fd" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 38857 } } peers { permanent_uuid: "063799ab6b434788be1447722615f67d" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 36953 } } peers { permanent_uuid: "c99f2a7a833f49c68022c8a09e09c9ef" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 44587 } } }
I20260925 22:36:34.791455 15188 tablet_replica.cc:442] T 057ccfc562754c5fa2b789c8056e05bf P e35ce5c2bbc9406e86e15892c07a97fd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:36:34.793181 15191 tablet_replica.cc:439] T 057ccfc562754c5fa2b789c8056e05bf P 063799ab6b434788be1447722615f67d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "063799ab6b434788be1447722615f67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35ce5c2bbc9406e86e15892c07a97fd" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 38857 } } peers { permanent_uuid: "063799ab6b434788be1447722615f67d" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 36953 } } peers { permanent_uuid: "c99f2a7a833f49c68022c8a09e09c9ef" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 44587 } } }
I20260925 22:36:34.793289 15191 tablet_replica.cc:442] T 057ccfc562754c5fa2b789c8056e05bf P 063799ab6b434788be1447722615f67d: This TxnStatusTablet replica's current role is: LEADER
I20260925 22:36:34.793475 15187 tablet_replica.cc:439] T 057ccfc562754c5fa2b789c8056e05bf P c99f2a7a833f49c68022c8a09e09c9ef: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "063799ab6b434788be1447722615f67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35ce5c2bbc9406e86e15892c07a97fd" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 38857 } } peers { permanent_uuid: "063799ab6b434788be1447722615f67d" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 36953 } } peers { permanent_uuid: "c99f2a7a833f49c68022c8a09e09c9ef" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 44587 } } }
I20260925 22:36:34.793581 15187 tablet_replica.cc:442] T 057ccfc562754c5fa2b789c8056e05bf P c99f2a7a833f49c68022c8a09e09c9ef: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:36:34.793885 15194 mvcc.cc:205] Tried to move back new op lower bound from 7333379255452745728 to 7333379255307698176. Current Snapshot: MvccSnapshot[applied={T|T < 7333379255452745728 or (T in {7333379255452745728})}]
I20260925 22:36:34.796422 15199 tablet_replica.cc:439] T 057ccfc562754c5fa2b789c8056e05bf P 063799ab6b434788be1447722615f67d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "063799ab6b434788be1447722615f67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35ce5c2bbc9406e86e15892c07a97fd" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 38857 } } peers { permanent_uuid: "063799ab6b434788be1447722615f67d" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 36953 } } peers { permanent_uuid: "c99f2a7a833f49c68022c8a09e09c9ef" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 44587 } } }
I20260925 22:36:34.796540 15199 tablet_replica.cc:442] T 057ccfc562754c5fa2b789c8056e05bf P 063799ab6b434788be1447722615f67d: This TxnStatusTablet replica's current role is: LEADER
I20260925 22:36:34.797245 15188 tablet_replica.cc:439] T 057ccfc562754c5fa2b789c8056e05bf P e35ce5c2bbc9406e86e15892c07a97fd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "063799ab6b434788be1447722615f67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35ce5c2bbc9406e86e15892c07a97fd" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 38857 } } peers { permanent_uuid: "063799ab6b434788be1447722615f67d" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 36953 } } peers { permanent_uuid: "c99f2a7a833f49c68022c8a09e09c9ef" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 44587 } } }
I20260925 22:36:34.797348 15188 tablet_replica.cc:442] T 057ccfc562754c5fa2b789c8056e05bf P e35ce5c2bbc9406e86e15892c07a97fd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:36:34.807729 15178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:34.804178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.194:49257 (local address 127.6.246.194:36953)
0925 22:36:34.804287 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:34.804291 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:34.804546 (+ 255us) server_negotiation.cc:407] Connection header received
0925 22:36:34.804666 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:34.804669 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:34.804706 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:34.804765 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:36:34.805037 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:34.805522 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:34.806705 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:34.806921 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:34.807062 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:34.807324 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:36:34.807398 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:36:34.807475 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:34.807549 (+ 74us) server_negotiation.cc:299] Negotiation successful
0925 22:36:34.807590 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 22:36:34.807731 15228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:34.804067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.194:36953 (local address 127.6.246.194:49257)
0925 22:36:34.804437 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:34.804474 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:34.804578 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:34.804797 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:34.804800 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:34.804807 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 22:36:34.804914 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:34.804920 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:34.805626 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:34.805630 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:34.806581 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:34.806591 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:34.807069 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:34.807072 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:34.807155 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:34.807189 (+ 34us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 22:36:34.807475 (+ 286us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 22:36:34.807481 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 22:36:34.807542 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 22:36:34.807617 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"mutex_wait_us":3,"thread_start_us":112,"threads_started":1}
I20260925 22:36:34.813660 14827 raft_consensus.cc:1261] T b2b2870876a5435eb57390f706dc7364 P e35ce5c2bbc9406e86e15892c07a97fd [term 1 FOLLOWER]: Refusing update from remote peer 063799ab6b434788be1447722615f67d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:36:34.813654 15107 raft_consensus.cc:1261] T b2b2870876a5435eb57390f706dc7364 P c99f2a7a833f49c68022c8a09e09c9ef [term 1 FOLLOWER]: Refusing update from remote peer 063799ab6b434788be1447722615f67d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:36:34.814035 15189 consensus_queue.cc:1035] T b2b2870876a5435eb57390f706dc7364 P 063799ab6b434788be1447722615f67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e35ce5c2bbc9406e86e15892c07a97fd" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 38857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:36:34.814167 15191 consensus_queue.cc:1035] T b2b2870876a5435eb57390f706dc7364 P 063799ab6b434788be1447722615f67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c99f2a7a833f49c68022c8a09e09c9ef" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 44587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:36:34.820469 15195 mvcc.cc:205] Tried to move back new op lower bound from 7333379255553466368 to 7333379254591336448. Current Snapshot: MvccSnapshot[applied={T|T < 7333379255553466368}]
I20260925 22:36:34.852072 14669 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:36:34.852241 14669 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:36:34.854223 14669 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 302768cab249462291567a64833dd1a1: Sending DeleteTablet for 3 replicas of tablet b2b2870876a5435eb57390f706dc7364
I20260925 22:36:34.854854 14946 tablet_service.cc:1514] Processing DeleteTablet for tablet b2b2870876a5435eb57390f706dc7364 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:36:34 UTC) from {username='oryx', principal='oryx/127.6.246.254@KRBTEST.COM'} at 127.0.0.1:45972
I20260925 22:36:34.854794 14807 tablet_service.cc:1514] Processing DeleteTablet for tablet b2b2870876a5435eb57390f706dc7364 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:36:34 UTC) from {username='oryx', principal='oryx/127.6.246.254@KRBTEST.COM'} at 127.0.0.1:58542
I20260925 22:36:34.855018 15205 tablet_replica.cc:330] T b2b2870876a5435eb57390f706dc7364 P e35ce5c2bbc9406e86e15892c07a97fd: stopping tablet replica
I20260925 22:36:34.855018 15204 tablet_replica.cc:330] T b2b2870876a5435eb57390f706dc7364 P 063799ab6b434788be1447722615f67d: stopping tablet replica
I20260925 22:36:34.855124 15204 raft_consensus.cc:2229] T b2b2870876a5435eb57390f706dc7364 P 063799ab6b434788be1447722615f67d [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:36:34.855122 15205 raft_consensus.cc:2229] T b2b2870876a5435eb57390f706dc7364 P e35ce5c2bbc9406e86e15892c07a97fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:36:34.855223 15205 raft_consensus.cc:2258] T b2b2870876a5435eb57390f706dc7364 P e35ce5c2bbc9406e86e15892c07a97fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:36:34.855285 15204 raft_consensus.cc:2258] T b2b2870876a5435eb57390f706dc7364 P 063799ab6b434788be1447722615f67d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:36:34.855914 15087 tablet_service.cc:1514] Processing DeleteTablet for tablet b2b2870876a5435eb57390f706dc7364 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:36:34 UTC) from {username='oryx', principal='oryx/127.6.246.254@KRBTEST.COM'} at 127.0.0.1:54576
I20260925 22:36:34.856087 15206 tablet_replica.cc:330] T b2b2870876a5435eb57390f706dc7364 P c99f2a7a833f49c68022c8a09e09c9ef: stopping tablet replica
I20260925 22:36:34.856223 15206 raft_consensus.cc:2229] T b2b2870876a5435eb57390f706dc7364 P c99f2a7a833f49c68022c8a09e09c9ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:36:34.856346 15206 raft_consensus.cc:2258] T b2b2870876a5435eb57390f706dc7364 P c99f2a7a833f49c68022c8a09e09c9ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:36:34.856560 15205 ts_tablet_manager.cc:1905] T b2b2870876a5435eb57390f706dc7364 P e35ce5c2bbc9406e86e15892c07a97fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:36:34.857040 15206 ts_tablet_manager.cc:1905] T b2b2870876a5435eb57390f706dc7364 P c99f2a7a833f49c68022c8a09e09c9ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:36:34.859573 15205 ts_tablet_manager.cc:1918] T b2b2870876a5435eb57390f706dc7364 P e35ce5c2bbc9406e86e15892c07a97fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 22:36:34.859666 15205 log.cc:1196] T b2b2870876a5435eb57390f706dc7364 P e35ce5c2bbc9406e86e15892c07a97fd: Deleting WAL directory at /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/minicluster-data/ts-0/wal/wals/b2b2870876a5435eb57390f706dc7364
I20260925 22:36:34.859764 15206 ts_tablet_manager.cc:1918] T b2b2870876a5435eb57390f706dc7364 P c99f2a7a833f49c68022c8a09e09c9ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 22:36:34.859843 15206 log.cc:1196] T b2b2870876a5435eb57390f706dc7364 P c99f2a7a833f49c68022c8a09e09c9ef: Deleting WAL directory at /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/minicluster-data/ts-2/wal/wals/b2b2870876a5435eb57390f706dc7364
I20260925 22:36:34.859961 15205 ts_tablet_manager.cc:1939] T b2b2870876a5435eb57390f706dc7364 P e35ce5c2bbc9406e86e15892c07a97fd: Deleting consensus metadata
I20260925 22:36:34.860095 15206 ts_tablet_manager.cc:1939] T b2b2870876a5435eb57390f706dc7364 P c99f2a7a833f49c68022c8a09e09c9ef: Deleting consensus metadata
I20260925 22:36:34.860689 14654 catalog_manager.cc:4841] TS e35ce5c2bbc9406e86e15892c07a97fd (127.6.246.193:38857): tablet b2b2870876a5435eb57390f706dc7364 (table test-table [id=6e724673d52547d4a077daf0c58842fc]) successfully deleted
I20260925 22:36:34.860888 14654 catalog_manager.cc:4841] TS c99f2a7a833f49c68022c8a09e09c9ef (127.6.246.195:44587): tablet b2b2870876a5435eb57390f706dc7364 (table test-table [id=6e724673d52547d4a077daf0c58842fc]) successfully deleted
I20260925 22:36:34.861347 15204 ts_tablet_manager.cc:1905] T b2b2870876a5435eb57390f706dc7364 P 063799ab6b434788be1447722615f67d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:36:34.863399 15204 ts_tablet_manager.cc:1918] T b2b2870876a5435eb57390f706dc7364 P 063799ab6b434788be1447722615f67d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 22:36:34.863468 15204 log.cc:1196] T b2b2870876a5435eb57390f706dc7364 P 063799ab6b434788be1447722615f67d: Deleting WAL directory at /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/minicluster-data/ts-1/wal/wals/b2b2870876a5435eb57390f706dc7364
I20260925 22:36:34.863713 15204 ts_tablet_manager.cc:1939] T b2b2870876a5435eb57390f706dc7364 P 063799ab6b434788be1447722615f67d: Deleting consensus metadata
I20260925 22:36:34.864351 14655 catalog_manager.cc:4841] TS 063799ab6b434788be1447722615f67d (127.6.246.194:36953): tablet b2b2870876a5435eb57390f706dc7364 (table test-table [id=6e724673d52547d4a077daf0c58842fc]) successfully deleted
Sep 25 22:36:34 dist-test-slave-wjdr krb5kdc[14625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375794, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 22:36:34 dist-test-slave-wjdr krb5kdc[14625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:36:34 dist-test-slave-wjdr krb5kdc[14625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
W20260925 22:36:34.889477 14719 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:36:34.883290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50346 (local address 127.6.246.254:38821)
0925 22:36:34.883420 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:34.883426 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:34.883595 (+ 169us) server_negotiation.cc:407] Connection header received
0925 22:36:34.883776 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:34.883778 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:34.883823 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:34.883938 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:34.884709 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:34.885228 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:34.886076 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:34.886297 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:34.886410 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:34.889336 (+ 2926us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50346: BlockingRecv error: recv got EOF from 127.0.0.1:50346 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 25 22:36:34 dist-test-slave-wjdr krb5kdc[14625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375794, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM
I20260925 22:36:34.899976 14719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:34.891055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50352 (local address 127.6.246.254:38821)
0925 22:36:34.891186 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:34.891189 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:34.891322 (+ 133us) server_negotiation.cc:407] Connection header received
0925 22:36:34.891438 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:34.891440 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:34.891476 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:34.891538 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:34.892293 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:34.892881 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:34.894015 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:34.894211 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:34.894308 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:34.896899 (+ 2591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:34.896902 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:34.896904 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:34.896926 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:34.898374 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:34.898784 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:34.898789 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:34.898791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:34.898894 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:34.899156 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:34.899161 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:34.899163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:34.899330 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:34.899466 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:34.899701 (+ 235us) server_negotiation.cc:299] Negotiation successful
0925 22:36:34.899814 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 22:36:34.902503 14669 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50352:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:36:34.903004 14669 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:36:34.907538 14807 tablet_service.cc:1467] Processing CreateTablet for tablet 344e83c273064744b6ee0bd6d201e111 (DEFAULT_TABLE table=test-table [id=d50e1a8c06c048b094978d7c2e13c8e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:36:34.907768 15087 tablet_service.cc:1467] Processing CreateTablet for tablet 344e83c273064744b6ee0bd6d201e111 (DEFAULT_TABLE table=test-table [id=d50e1a8c06c048b094978d7c2e13c8e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:36:34.907842 14807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 344e83c273064744b6ee0bd6d201e111. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:36:34.908075 15087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 344e83c273064744b6ee0bd6d201e111. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:36:34.908996 14946 tablet_service.cc:1467] Processing CreateTablet for tablet 344e83c273064744b6ee0bd6d201e111 (DEFAULT_TABLE table=test-table [id=d50e1a8c06c048b094978d7c2e13c8e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:36:34.909257 14946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 344e83c273064744b6ee0bd6d201e111. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:36:34.910435 15181 tablet_bootstrap.cc:492] T 344e83c273064744b6ee0bd6d201e111 P c99f2a7a833f49c68022c8a09e09c9ef: Bootstrap starting.
I20260925 22:36:34.911259 15181 tablet_bootstrap.cc:654] T 344e83c273064744b6ee0bd6d201e111 P c99f2a7a833f49c68022c8a09e09c9ef: Neither blocks nor log segments found. Creating new log.
I20260925 22:36:34.913384 15184 tablet_bootstrap.cc:492] T 344e83c273064744b6ee0bd6d201e111 P e35ce5c2bbc9406e86e15892c07a97fd: Bootstrap starting.
I20260925 22:36:34.913964 15182 tablet_bootstrap.cc:492] T 344e83c273064744b6ee0bd6d201e111 P 063799ab6b434788be1447722615f67d: Bootstrap starting.
I20260925 22:36:34.914369 15184 tablet_bootstrap.cc:654] T 344e83c273064744b6ee0bd6d201e111 P e35ce5c2bbc9406e86e15892c07a97fd: Neither blocks nor log segments found. Creating new log.
I20260925 22:36:34.914762 15181 tablet_bootstrap.cc:492] T 344e83c273064744b6ee0bd6d201e111 P c99f2a7a833f49c68022c8a09e09c9ef: No bootstrap required, opened a new log
I20260925 22:36:34.914834 15181 ts_tablet_manager.cc:1397] T 344e83c273064744b6ee0bd6d201e111 P c99f2a7a833f49c68022c8a09e09c9ef: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260925 22:36:34.914849 15182 tablet_bootstrap.cc:654] T 344e83c273064744b6ee0bd6d201e111 P 063799ab6b434788be1447722615f67d: Neither blocks nor log segments found. Creating new log.
I20260925 22:36:34.915256 15181 raft_consensus.cc:348] T 344e83c273064744b6ee0bd6d201e111 P c99f2a7a833f49c68022c8a09e09c9ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35ce5c2bbc9406e86e15892c07a97fd" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 38857 } } peers { permanent_uuid: "063799ab6b434788be1447722615f67d" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 36953 } } peers { permanent_uuid: "c99f2a7a833f49c68022c8a09e09c9ef" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 44587 } }
I20260925 22:36:34.915360 15181 raft_consensus.cc:374] T 344e83c273064744b6ee0bd6d201e111 P c99f2a7a833f49c68022c8a09e09c9ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:36:34.915388 15181 raft_consensus.cc:729] T 344e83c273064744b6ee0bd6d201e111 P c99f2a7a833f49c68022c8a09e09c9ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c99f2a7a833f49c68022c8a09e09c9ef, State: Initialized, Role: FOLLOWER
I20260925 22:36:34.915510 15181 consensus_queue.cc:260] T 344e83c273064744b6ee0bd6d201e111 P c99f2a7a833f49c68022c8a09e09c9ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35ce5c2bbc9406e86e15892c07a97fd" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 38857 } } peers { permanent_uuid: "063799ab6b434788be1447722615f67d" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 36953 } } peers { permanent_uuid: "c99f2a7a833f49c68022c8a09e09c9ef" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 44587 } }
I20260925 22:36:34.915670 15184 tablet_bootstrap.cc:492] T 344e83c273064744b6ee0bd6d201e111 P e35ce5c2bbc9406e86e15892c07a97fd: No bootstrap required, opened a new log
I20260925 22:36:34.915748 15181 ts_tablet_manager.cc:1428] T 344e83c273064744b6ee0bd6d201e111 P c99f2a7a833f49c68022c8a09e09c9ef: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260925 22:36:34.915750 15184 ts_tablet_manager.cc:1397] T 344e83c273064744b6ee0bd6d201e111 P e35ce5c2bbc9406e86e15892c07a97fd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 22:36:34.916213 15184 raft_consensus.cc:348] T 344e83c273064744b6ee0bd6d201e111 P e35ce5c2bbc9406e86e15892c07a97fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35ce5c2bbc9406e86e15892c07a97fd" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 38857 } } peers { permanent_uuid: "063799ab6b434788be1447722615f67d" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 36953 } } peers { permanent_uuid: "c99f2a7a833f49c68022c8a09e09c9ef" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 44587 } }
I20260925 22:36:34.916321 15184 raft_consensus.cc:374] T 344e83c273064744b6ee0bd6d201e111 P e35ce5c2bbc9406e86e15892c07a97fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:36:34.916352 15184 raft_consensus.cc:729] T 344e83c273064744b6ee0bd6d201e111 P e35ce5c2bbc9406e86e15892c07a97fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e35ce5c2bbc9406e86e15892c07a97fd, State: Initialized, Role: FOLLOWER
I20260925 22:36:34.916955 15182 tablet_bootstrap.cc:492] T 344e83c273064744b6ee0bd6d201e111 P 063799ab6b434788be1447722615f67d: No bootstrap required, opened a new log
I20260925 22:36:34.917038 15182 ts_tablet_manager.cc:1397] T 344e83c273064744b6ee0bd6d201e111 P 063799ab6b434788be1447722615f67d: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260925 22:36:34.917021 15184 consensus_queue.cc:260] T 344e83c273064744b6ee0bd6d201e111 P e35ce5c2bbc9406e86e15892c07a97fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35ce5c2bbc9406e86e15892c07a97fd" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 38857 } } peers { permanent_uuid: "063799ab6b434788be1447722615f67d" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 36953 } } peers { permanent_uuid: "c99f2a7a833f49c68022c8a09e09c9ef" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 44587 } }
I20260925 22:36:34.917300 15184 ts_tablet_manager.cc:1428] T 344e83c273064744b6ee0bd6d201e111 P e35ce5c2bbc9406e86e15892c07a97fd: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260925 22:36:34.917404 15182 raft_consensus.cc:348] T 344e83c273064744b6ee0bd6d201e111 P 063799ab6b434788be1447722615f67d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35ce5c2bbc9406e86e15892c07a97fd" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 38857 } } peers { permanent_uuid: "063799ab6b434788be1447722615f67d" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 36953 } } peers { permanent_uuid: "c99f2a7a833f49c68022c8a09e09c9ef" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 44587 } }
I20260925 22:36:34.917497 15182 raft_consensus.cc:374] T 344e83c273064744b6ee0bd6d201e111 P 063799ab6b434788be1447722615f67d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:36:34.917523 15182 raft_consensus.cc:729] T 344e83c273064744b6ee0bd6d201e111 P 063799ab6b434788be1447722615f67d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 063799ab6b434788be1447722615f67d, State: Initialized, Role: FOLLOWER
I20260925 22:36:34.917625 15182 consensus_queue.cc:260] T 344e83c273064744b6ee0bd6d201e111 P 063799ab6b434788be1447722615f67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35ce5c2bbc9406e86e15892c07a97fd" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 38857 } } peers { permanent_uuid: "063799ab6b434788be1447722615f67d" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 36953 } } peers { permanent_uuid: "c99f2a7a833f49c68022c8a09e09c9ef" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 44587 } }
I20260925 22:36:34.917835 15182 ts_tablet_manager.cc:1428] T 344e83c273064744b6ee0bd6d201e111 P 063799ab6b434788be1447722615f67d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 22:36:35.036728 15188 raft_consensus.cc:482] T 344e83c273064744b6ee0bd6d201e111 P e35ce5c2bbc9406e86e15892c07a97fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:36:35.036864 15188 raft_consensus.cc:504] T 344e83c273064744b6ee0bd6d201e111 P e35ce5c2bbc9406e86e15892c07a97fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35ce5c2bbc9406e86e15892c07a97fd" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 38857 } } peers { permanent_uuid: "063799ab6b434788be1447722615f67d" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 36953 } } peers { permanent_uuid: "c99f2a7a833f49c68022c8a09e09c9ef" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 44587 } }
I20260925 22:36:35.038009 15188 leader_election.cc:288] T 344e83c273064744b6ee0bd6d201e111 P e35ce5c2bbc9406e86e15892c07a97fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 063799ab6b434788be1447722615f67d (127.6.246.194:36953), c99f2a7a833f49c68022c8a09e09c9ef (127.6.246.195:44587)
I20260925 22:36:35.041931 15178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:35.038365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.193:58239 (local address 127.6.246.194:36953)
0925 22:36:35.038501 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:35.038507 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:35.038680 (+ 173us) server_negotiation.cc:407] Connection header received
0925 22:36:35.038837 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:35.038845 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:35.038888 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:35.038975 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:36:35.039208 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:35.039711 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:35.040986 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:35.041389 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:35.041518 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:35.041573 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:35.041733 (+ 160us) server_negotiation.cc:299] Negotiation successful
0925 22:36:35.041802 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 22:36:35.041951 15248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:35.038233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.194:36953 (local address 127.6.246.193:58239)
0925 22:36:35.038570 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:35.038584 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:35.038717 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:35.038987 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:35.038989 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:35.039005 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:36:35.039113 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:35.039120 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:35.039894 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:35.039898 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:35.040859 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:35.040868 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:35.041510 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:35.041514 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:35.041600 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:35.041630 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 22:36:35.041717 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 22:36:35.041785 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":101,"threads_started":1}
I20260925 22:36:35.042676 14967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "344e83c273064744b6ee0bd6d201e111" candidate_uuid: "e35ce5c2bbc9406e86e15892c07a97fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "063799ab6b434788be1447722615f67d" is_pre_election: true
I20260925 22:36:35.042881 15247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:35.038274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.195:44587 (local address 127.6.246.193:55421)
0925 22:36:35.038722 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:35.038739 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:35.038878 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:35.039171 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:35.039174 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:35.039189 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:36:35.039327 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:35.039333 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:35.040251 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:35.040257 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:35.041018 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:35.041027 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:35.041681 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:35.041688 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:35.042564 (+ 876us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:35.042589 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 22:36:35.042671 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 22:36:35.042742 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":66,"threads_started":1}
I20260925 22:36:35.042943 14967 raft_consensus.cc:2454] T 344e83c273064744b6ee0bd6d201e111 P 063799ab6b434788be1447722615f67d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e35ce5c2bbc9406e86e15892c07a97fd in term 0.
I20260925 22:36:35.043161 15249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:35.038348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.193:55421 (local address 127.6.246.195:44587)
0925 22:36:35.038725 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:35.038731 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:35.038824 (+ 93us) server_negotiation.cc:407] Connection header received
0925 22:36:35.039024 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:35.039026 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:35.039068 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:35.039163 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:36:35.039443 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:35.040143 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:35.041142 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:35.041550 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:35.041696 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:35.041773 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:35.042695 (+ 922us) server_negotiation.cc:299] Negotiation successful
0925 22:36:35.042759 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":89,"threads_started":1}
I20260925 22:36:35.043406 14759 leader_election.cc:302] T 344e83c273064744b6ee0bd6d201e111 P e35ce5c2bbc9406e86e15892c07a97fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 063799ab6b434788be1447722615f67d, e35ce5c2bbc9406e86e15892c07a97fd; no voters:
I20260925 22:36:35.043515 15107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "344e83c273064744b6ee0bd6d201e111" candidate_uuid: "e35ce5c2bbc9406e86e15892c07a97fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c99f2a7a833f49c68022c8a09e09c9ef" is_pre_election: true
I20260925 22:36:35.043672 15107 raft_consensus.cc:2454] T 344e83c273064744b6ee0bd6d201e111 P c99f2a7a833f49c68022c8a09e09c9ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e35ce5c2bbc9406e86e15892c07a97fd in term 0.
I20260925 22:36:35.043687 15188 raft_consensus.cc:2781] T 344e83c273064744b6ee0bd6d201e111 P e35ce5c2bbc9406e86e15892c07a97fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:36:35.043746 15188 raft_consensus.cc:482] T 344e83c273064744b6ee0bd6d201e111 P e35ce5c2bbc9406e86e15892c07a97fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:36:35.043807 15188 raft_consensus.cc:3037] T 344e83c273064744b6ee0bd6d201e111 P e35ce5c2bbc9406e86e15892c07a97fd [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:36:35.044874 15188 raft_consensus.cc:504] T 344e83c273064744b6ee0bd6d201e111 P e35ce5c2bbc9406e86e15892c07a97fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35ce5c2bbc9406e86e15892c07a97fd" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 38857 } } peers { permanent_uuid: "063799ab6b434788be1447722615f67d" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 36953 } } peers { permanent_uuid: "c99f2a7a833f49c68022c8a09e09c9ef" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 44587 } }
I20260925 22:36:35.045262 15188 leader_election.cc:288] T 344e83c273064744b6ee0bd6d201e111 P e35ce5c2bbc9406e86e15892c07a97fd [CANDIDATE]: Term 1 election: Requested vote from peers 063799ab6b434788be1447722615f67d (127.6.246.194:36953), c99f2a7a833f49c68022c8a09e09c9ef (127.6.246.195:44587)
I20260925 22:36:35.045576 14967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "344e83c273064744b6ee0bd6d201e111" candidate_uuid: "e35ce5c2bbc9406e86e15892c07a97fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "063799ab6b434788be1447722615f67d"
I20260925 22:36:35.045629 15107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "344e83c273064744b6ee0bd6d201e111" candidate_uuid: "e35ce5c2bbc9406e86e15892c07a97fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c99f2a7a833f49c68022c8a09e09c9ef"
I20260925 22:36:35.045723 14967 raft_consensus.cc:3037] T 344e83c273064744b6ee0bd6d201e111 P 063799ab6b434788be1447722615f67d [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:36:35.045745 15107 raft_consensus.cc:3037] T 344e83c273064744b6ee0bd6d201e111 P c99f2a7a833f49c68022c8a09e09c9ef [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:36:35.046639 15107 raft_consensus.cc:2454] T 344e83c273064744b6ee0bd6d201e111 P c99f2a7a833f49c68022c8a09e09c9ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e35ce5c2bbc9406e86e15892c07a97fd in term 1.
I20260925 22:36:35.046646 14967 raft_consensus.cc:2454] T 344e83c273064744b6ee0bd6d201e111 P 063799ab6b434788be1447722615f67d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e35ce5c2bbc9406e86e15892c07a97fd in term 1.
I20260925 22:36:35.046924 14759 leader_election.cc:302] T 344e83c273064744b6ee0bd6d201e111 P e35ce5c2bbc9406e86e15892c07a97fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 063799ab6b434788be1447722615f67d, e35ce5c2bbc9406e86e15892c07a97fd; no voters:
I20260925 22:36:35.047123 15188 raft_consensus.cc:2781] T 344e83c273064744b6ee0bd6d201e111 P e35ce5c2bbc9406e86e15892c07a97fd [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:36:35.047395 15188 raft_consensus.cc:686] T 344e83c273064744b6ee0bd6d201e111 P e35ce5c2bbc9406e86e15892c07a97fd [term 1 LEADER]: Becoming Leader. State: Replica: e35ce5c2bbc9406e86e15892c07a97fd, State: Running, Role: LEADER
I20260925 22:36:35.047659 15188 consensus_queue.cc:237] T 344e83c273064744b6ee0bd6d201e111 P e35ce5c2bbc9406e86e15892c07a97fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35ce5c2bbc9406e86e15892c07a97fd" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 38857 } } peers { permanent_uuid: "063799ab6b434788be1447722615f67d" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 36953 } } peers { permanent_uuid: "c99f2a7a833f49c68022c8a09e09c9ef" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 44587 } }
I20260925 22:36:35.049985 14669 catalog_manager.cc:5495] T 344e83c273064744b6ee0bd6d201e111 P e35ce5c2bbc9406e86e15892c07a97fd reported cstate change: term changed from 0 to 1, leader changed from <none> to e35ce5c2bbc9406e86e15892c07a97fd (127.6.246.193). New cstate: current_term: 1 leader_uuid: "e35ce5c2bbc9406e86e15892c07a97fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35ce5c2bbc9406e86e15892c07a97fd" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 38857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "063799ab6b434788be1447722615f67d" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 36953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c99f2a7a833f49c68022c8a09e09c9ef" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 44587 } health_report { overall_health: UNKNOWN } } }
I20260925 22:36:35.067631 15253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:35.063991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58558 (local address 127.6.246.193:38857)
0925 22:36:35.064256 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:35.064275 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:35.064286 (+ 11us) server_negotiation.cc:407] Connection header received
0925 22:36:35.064365 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:35.064367 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:35.064416 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:35.064586 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:36:35.064827 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:35.065368 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:35.066212 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:35.066446 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:35.066549 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:35.066814 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:36:35.067248 (+ 434us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:36:35.067381 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:35.067421 (+ 40us) server_negotiation.cc:299] Negotiation successful
0925 22:36:35.067471 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":86,"threads_started":1}
I20260925 22:36:35.070266 15107 raft_consensus.cc:1261] T 344e83c273064744b6ee0bd6d201e111 P c99f2a7a833f49c68022c8a09e09c9ef [term 1 FOLLOWER]: Refusing update from remote peer e35ce5c2bbc9406e86e15892c07a97fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:36:35.070429 14967 raft_consensus.cc:1261] T 344e83c273064744b6ee0bd6d201e111 P 063799ab6b434788be1447722615f67d [term 1 FOLLOWER]: Refusing update from remote peer e35ce5c2bbc9406e86e15892c07a97fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:36:35.070770 15250 consensus_queue.cc:1035] T 344e83c273064744b6ee0bd6d201e111 P e35ce5c2bbc9406e86e15892c07a97fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c99f2a7a833f49c68022c8a09e09c9ef" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 44587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:36:35.071008 15188 consensus_queue.cc:1035] T 344e83c273064744b6ee0bd6d201e111 P e35ce5c2bbc9406e86e15892c07a97fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "063799ab6b434788be1447722615f67d" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 36953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:36:35.082486 14669 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:36:35.082635 14669 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:36:35.084627 14669 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 302768cab249462291567a64833dd1a1: Sending DeleteTablet for 3 replicas of tablet 344e83c273064744b6ee0bd6d201e111
I20260925 22:36:35.085254 14807 tablet_service.cc:1514] Processing DeleteTablet for tablet 344e83c273064744b6ee0bd6d201e111 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:36:35 UTC) from {username='oryx', principal='oryx/127.6.246.254@KRBTEST.COM'} at 127.0.0.1:58542
I20260925 22:36:35.085245 14946 tablet_service.cc:1514] Processing DeleteTablet for tablet 344e83c273064744b6ee0bd6d201e111 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:36:35 UTC) from {username='oryx', principal='oryx/127.6.246.254@KRBTEST.COM'} at 127.0.0.1:45972
I20260925 22:36:35.085254 15087 tablet_service.cc:1514] Processing DeleteTablet for tablet 344e83c273064744b6ee0bd6d201e111 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:36:35 UTC) from {username='oryx', principal='oryx/127.6.246.254@KRBTEST.COM'} at 127.0.0.1:54576
I20260925 22:36:35.085419 15204 tablet_replica.cc:330] T 344e83c273064744b6ee0bd6d201e111 P 063799ab6b434788be1447722615f67d: stopping tablet replica
I20260925 22:36:35.085444 15205 tablet_replica.cc:330] T 344e83c273064744b6ee0bd6d201e111 P e35ce5c2bbc9406e86e15892c07a97fd: stopping tablet replica
I20260925 22:36:35.085538 15204 raft_consensus.cc:2229] T 344e83c273064744b6ee0bd6d201e111 P 063799ab6b434788be1447722615f67d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:36:35.085546 15205 raft_consensus.cc:2229] T 344e83c273064744b6ee0bd6d201e111 P e35ce5c2bbc9406e86e15892c07a97fd [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:36:35.085670 15204 raft_consensus.cc:2258] T 344e83c273064744b6ee0bd6d201e111 P 063799ab6b434788be1447722615f67d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:36:35.085810 15205 raft_consensus.cc:2258] T 344e83c273064744b6ee0bd6d201e111 P e35ce5c2bbc9406e86e15892c07a97fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:36:35.085992 15206 tablet_replica.cc:330] T 344e83c273064744b6ee0bd6d201e111 P c99f2a7a833f49c68022c8a09e09c9ef: stopping tablet replica
I20260925 22:36:35.086148 15206 raft_consensus.cc:2229] T 344e83c273064744b6ee0bd6d201e111 P c99f2a7a833f49c68022c8a09e09c9ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:36:35.086267 15206 raft_consensus.cc:2258] T 344e83c273064744b6ee0bd6d201e111 P c99f2a7a833f49c68022c8a09e09c9ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:36:35.086316 15204 ts_tablet_manager.cc:1905] T 344e83c273064744b6ee0bd6d201e111 P 063799ab6b434788be1447722615f67d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:36:35.087088 15205 ts_tablet_manager.cc:1905] T 344e83c273064744b6ee0bd6d201e111 P e35ce5c2bbc9406e86e15892c07a97fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:36:35.087205 14669 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50352:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:36:35.087592 14669 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:36:35.089322 15206 ts_tablet_manager.cc:1905] T 344e83c273064744b6ee0bd6d201e111 P c99f2a7a833f49c68022c8a09e09c9ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:36:35.089509 15205 ts_tablet_manager.cc:1918] T 344e83c273064744b6ee0bd6d201e111 P e35ce5c2bbc9406e86e15892c07a97fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:36:35.089578 15205 log.cc:1196] T 344e83c273064744b6ee0bd6d201e111 P e35ce5c2bbc9406e86e15892c07a97fd: Deleting WAL directory at /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/minicluster-data/ts-0/wal/wals/344e83c273064744b6ee0bd6d201e111
I20260925 22:36:35.089910 15205 ts_tablet_manager.cc:1939] T 344e83c273064744b6ee0bd6d201e111 P e35ce5c2bbc9406e86e15892c07a97fd: Deleting consensus metadata
I20260925 22:36:35.090535 14654 catalog_manager.cc:4841] TS e35ce5c2bbc9406e86e15892c07a97fd (127.6.246.193:38857): tablet 344e83c273064744b6ee0bd6d201e111 (table test-table [id=d50e1a8c06c048b094978d7c2e13c8e8]) successfully deleted
I20260925 22:36:35.091732 15206 ts_tablet_manager.cc:1918] T 344e83c273064744b6ee0bd6d201e111 P c99f2a7a833f49c68022c8a09e09c9ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:36:35.091806 15206 log.cc:1196] T 344e83c273064744b6ee0bd6d201e111 P c99f2a7a833f49c68022c8a09e09c9ef: Deleting WAL directory at /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/minicluster-data/ts-2/wal/wals/344e83c273064744b6ee0bd6d201e111
I20260925 22:36:35.092077 15206 ts_tablet_manager.cc:1939] T 344e83c273064744b6ee0bd6d201e111 P c99f2a7a833f49c68022c8a09e09c9ef: Deleting consensus metadata
I20260925 22:36:35.092562 14946 tablet_service.cc:1467] Processing CreateTablet for tablet 40d4c4a6ffe24789a5b9715ecc1960a8 (DEFAULT_TABLE table=test-table [id=50319928ebb14f1aab49eeef8532bd9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:36:35.092706 14654 catalog_manager.cc:4841] TS c99f2a7a833f49c68022c8a09e09c9ef (127.6.246.195:44587): tablet 344e83c273064744b6ee0bd6d201e111 (table test-table [id=d50e1a8c06c048b094978d7c2e13c8e8]) successfully deleted
I20260925 22:36:35.092717 15087 tablet_service.cc:1467] Processing CreateTablet for tablet 40d4c4a6ffe24789a5b9715ecc1960a8 (DEFAULT_TABLE table=test-table [id=50319928ebb14f1aab49eeef8532bd9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:36:35.092938 15087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40d4c4a6ffe24789a5b9715ecc1960a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:36:35.093147 15204 ts_tablet_manager.cc:1918] T 344e83c273064744b6ee0bd6d201e111 P 063799ab6b434788be1447722615f67d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:36:35.093248 15204 log.cc:1196] T 344e83c273064744b6ee0bd6d201e111 P 063799ab6b434788be1447722615f67d: Deleting WAL directory at /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/minicluster-data/ts-1/wal/wals/344e83c273064744b6ee0bd6d201e111
I20260925 22:36:35.093534 15204 ts_tablet_manager.cc:1939] T 344e83c273064744b6ee0bd6d201e111 P 063799ab6b434788be1447722615f67d: Deleting consensus metadata
I20260925 22:36:35.093747 14946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40d4c4a6ffe24789a5b9715ecc1960a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:36:35.095183 15181 tablet_bootstrap.cc:492] T 40d4c4a6ffe24789a5b9715ecc1960a8 P c99f2a7a833f49c68022c8a09e09c9ef: Bootstrap starting.
I20260925 22:36:35.095557 15182 tablet_bootstrap.cc:492] T 40d4c4a6ffe24789a5b9715ecc1960a8 P 063799ab6b434788be1447722615f67d: Bootstrap starting.
I20260925 22:36:35.096052 15181 tablet_bootstrap.cc:654] T 40d4c4a6ffe24789a5b9715ecc1960a8 P c99f2a7a833f49c68022c8a09e09c9ef: Neither blocks nor log segments found. Creating new log.
I20260925 22:36:35.096267 14655 catalog_manager.cc:4841] TS 063799ab6b434788be1447722615f67d (127.6.246.194:36953): tablet 344e83c273064744b6ee0bd6d201e111 (table test-table [id=d50e1a8c06c048b094978d7c2e13c8e8]) successfully deleted
I20260925 22:36:35.096345 15182 tablet_bootstrap.cc:654] T 40d4c4a6ffe24789a5b9715ecc1960a8 P 063799ab6b434788be1447722615f67d: Neither blocks nor log segments found. Creating new log.
I20260925 22:36:35.097159 14807 tablet_service.cc:1467] Processing CreateTablet for tablet 40d4c4a6ffe24789a5b9715ecc1960a8 (DEFAULT_TABLE table=test-table [id=50319928ebb14f1aab49eeef8532bd9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:36:35.097242 15181 tablet_bootstrap.cc:492] T 40d4c4a6ffe24789a5b9715ecc1960a8 P c99f2a7a833f49c68022c8a09e09c9ef: No bootstrap required, opened a new log
I20260925 22:36:35.097308 15181 ts_tablet_manager.cc:1397] T 40d4c4a6ffe24789a5b9715ecc1960a8 P c99f2a7a833f49c68022c8a09e09c9ef: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 22:36:35.097424 14807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40d4c4a6ffe24789a5b9715ecc1960a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:36:35.097707 15181 raft_consensus.cc:348] T 40d4c4a6ffe24789a5b9715ecc1960a8 P c99f2a7a833f49c68022c8a09e09c9ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35ce5c2bbc9406e86e15892c07a97fd" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 38857 } } peers { permanent_uuid: "063799ab6b434788be1447722615f67d" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 36953 } } peers { permanent_uuid: "c99f2a7a833f49c68022c8a09e09c9ef" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 44587 } }
I20260925 22:36:35.097817 15181 raft_consensus.cc:374] T 40d4c4a6ffe24789a5b9715ecc1960a8 P c99f2a7a833f49c68022c8a09e09c9ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:36:35.097851 15181 raft_consensus.cc:729] T 40d4c4a6ffe24789a5b9715ecc1960a8 P c99f2a7a833f49c68022c8a09e09c9ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c99f2a7a833f49c68022c8a09e09c9ef, State: Initialized, Role: FOLLOWER
I20260925 22:36:35.097961 15181 consensus_queue.cc:260] T 40d4c4a6ffe24789a5b9715ecc1960a8 P c99f2a7a833f49c68022c8a09e09c9ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35ce5c2bbc9406e86e15892c07a97fd" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 38857 } } peers { permanent_uuid: "063799ab6b434788be1447722615f67d" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 36953 } } peers { permanent_uuid: "c99f2a7a833f49c68022c8a09e09c9ef" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 44587 } }
I20260925 22:36:35.098215 15181 ts_tablet_manager.cc:1428] T 40d4c4a6ffe24789a5b9715ecc1960a8 P c99f2a7a833f49c68022c8a09e09c9ef: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:36:35.099128 15182 tablet_bootstrap.cc:492] T 40d4c4a6ffe24789a5b9715ecc1960a8 P 063799ab6b434788be1447722615f67d: No bootstrap required, opened a new log
I20260925 22:36:35.099205 15182 ts_tablet_manager.cc:1397] T 40d4c4a6ffe24789a5b9715ecc1960a8 P 063799ab6b434788be1447722615f67d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260925 22:36:35.099604 15182 raft_consensus.cc:348] T 40d4c4a6ffe24789a5b9715ecc1960a8 P 063799ab6b434788be1447722615f67d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35ce5c2bbc9406e86e15892c07a97fd" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 38857 } } peers { permanent_uuid: "063799ab6b434788be1447722615f67d" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 36953 } } peers { permanent_uuid: "c99f2a7a833f49c68022c8a09e09c9ef" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 44587 } }
I20260925 22:36:35.099710 15182 raft_consensus.cc:374] T 40d4c4a6ffe24789a5b9715ecc1960a8 P 063799ab6b434788be1447722615f67d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:36:35.099713 15184 tablet_bootstrap.cc:492] T 40d4c4a6ffe24789a5b9715ecc1960a8 P e35ce5c2bbc9406e86e15892c07a97fd: Bootstrap starting.
I20260925 22:36:35.099743 15182 raft_consensus.cc:729] T 40d4c4a6ffe24789a5b9715ecc1960a8 P 063799ab6b434788be1447722615f67d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 063799ab6b434788be1447722615f67d, State: Initialized, Role: FOLLOWER
I20260925 22:36:35.099860 15182 consensus_queue.cc:260] T 40d4c4a6ffe24789a5b9715ecc1960a8 P 063799ab6b434788be1447722615f67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35ce5c2bbc9406e86e15892c07a97fd" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 38857 } } peers { permanent_uuid: "063799ab6b434788be1447722615f67d" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 36953 } } peers { permanent_uuid: "c99f2a7a833f49c68022c8a09e09c9ef" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 44587 } }
I20260925 22:36:35.100165 15182 ts_tablet_manager.cc:1428] T 40d4c4a6ffe24789a5b9715ecc1960a8 P 063799ab6b434788be1447722615f67d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:36:35.100556 15184 tablet_bootstrap.cc:654] T 40d4c4a6ffe24789a5b9715ecc1960a8 P e35ce5c2bbc9406e86e15892c07a97fd: Neither blocks nor log segments found. Creating new log.
I20260925 22:36:35.105185 15184 tablet_bootstrap.cc:492] T 40d4c4a6ffe24789a5b9715ecc1960a8 P e35ce5c2bbc9406e86e15892c07a97fd: No bootstrap required, opened a new log
I20260925 22:36:35.105279 15184 ts_tablet_manager.cc:1397] T 40d4c4a6ffe24789a5b9715ecc1960a8 P e35ce5c2bbc9406e86e15892c07a97fd: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20260925 22:36:35.105515 15187 raft_consensus.cc:482] T 40d4c4a6ffe24789a5b9715ecc1960a8 P c99f2a7a833f49c68022c8a09e09c9ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:36:35.105624 15187 raft_consensus.cc:504] T 40d4c4a6ffe24789a5b9715ecc1960a8 P c99f2a7a833f49c68022c8a09e09c9ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35ce5c2bbc9406e86e15892c07a97fd" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 38857 } } peers { permanent_uuid: "063799ab6b434788be1447722615f67d" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 36953 } } peers { permanent_uuid: "c99f2a7a833f49c68022c8a09e09c9ef" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 44587 } }
I20260925 22:36:35.105729 15184 raft_consensus.cc:348] T 40d4c4a6ffe24789a5b9715ecc1960a8 P e35ce5c2bbc9406e86e15892c07a97fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35ce5c2bbc9406e86e15892c07a97fd" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 38857 } } peers { permanent_uuid: "063799ab6b434788be1447722615f67d" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 36953 } } peers { permanent_uuid: "c99f2a7a833f49c68022c8a09e09c9ef" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 44587 } }
I20260925 22:36:35.105839 15184 raft_consensus.cc:374] T 40d4c4a6ffe24789a5b9715ecc1960a8 P e35ce5c2bbc9406e86e15892c07a97fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:36:35.105876 15184 raft_consensus.cc:729] T 40d4c4a6ffe24789a5b9715ecc1960a8 P e35ce5c2bbc9406e86e15892c07a97fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e35ce5c2bbc9406e86e15892c07a97fd, State: Initialized, Role: FOLLOWER
I20260925 22:36:35.105988 15184 consensus_queue.cc:260] T 40d4c4a6ffe24789a5b9715ecc1960a8 P e35ce5c2bbc9406e86e15892c07a97fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35ce5c2bbc9406e86e15892c07a97fd" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 38857 } } peers { permanent_uuid: "063799ab6b434788be1447722615f67d" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 36953 } } peers { permanent_uuid: "c99f2a7a833f49c68022c8a09e09c9ef" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 44587 } }
I20260925 22:36:35.106238 15184 ts_tablet_manager.cc:1428] T 40d4c4a6ffe24789a5b9715ecc1960a8 P e35ce5c2bbc9406e86e15892c07a97fd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 22:36:35.106978 15187 leader_election.cc:288] T 40d4c4a6ffe24789a5b9715ecc1960a8 P c99f2a7a833f49c68022c8a09e09c9ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e35ce5c2bbc9406e86e15892c07a97fd (127.6.246.193:38857), 063799ab6b434788be1447722615f67d (127.6.246.194:36953)
I20260925 22:36:35.111563 15253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:35.107487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.195:56475 (local address 127.6.246.193:38857)
0925 22:36:35.107675 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:35.107679 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:35.107693 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:36:35.107744 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:35.107747 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:35.107812 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:35.107883 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:36:35.108141 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:35.108803 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:35.110313 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:35.110833 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:35.110933 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:35.110990 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:35.111343 (+ 353us) server_negotiation.cc:299] Negotiation successful
0925 22:36:35.111433 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
I20260925 22:36:35.111976 15260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:35.107130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.193:38857 (local address 127.6.246.195:56475)
0925 22:36:35.107563 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:35.107578 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:35.107669 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:35.107907 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:35.107911 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:35.107935 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:36:35.108053 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:35.108059 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:35.108947 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:35.108950 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:35.110176 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:35.110187 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:35.111094 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:35.111098 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:35.111191 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:35.111216 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 22:36:35.111713 (+ 497us) client_negotiation.cc:240] Negotiation successful
0925 22:36:35.111848 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":84,"threads_started":1}
I20260925 22:36:35.112401 14827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40d4c4a6ffe24789a5b9715ecc1960a8" candidate_uuid: "c99f2a7a833f49c68022c8a09e09c9ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e35ce5c2bbc9406e86e15892c07a97fd" is_pre_election: true
I20260925 22:36:35.112578 14827 raft_consensus.cc:2454] T 40d4c4a6ffe24789a5b9715ecc1960a8 P e35ce5c2bbc9406e86e15892c07a97fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c99f2a7a833f49c68022c8a09e09c9ef in term 0.
I20260925 22:36:35.112964 15038 leader_election.cc:302] T 40d4c4a6ffe24789a5b9715ecc1960a8 P c99f2a7a833f49c68022c8a09e09c9ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c99f2a7a833f49c68022c8a09e09c9ef, e35ce5c2bbc9406e86e15892c07a97fd; no voters:
I20260925 22:36:35.113226 15187 raft_consensus.cc:2781] T 40d4c4a6ffe24789a5b9715ecc1960a8 P c99f2a7a833f49c68022c8a09e09c9ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:36:35.113287 15187 raft_consensus.cc:482] T 40d4c4a6ffe24789a5b9715ecc1960a8 P c99f2a7a833f49c68022c8a09e09c9ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:36:35.113333 15187 raft_consensus.cc:3037] T 40d4c4a6ffe24789a5b9715ecc1960a8 P c99f2a7a833f49c68022c8a09e09c9ef [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:36:35.113662 15261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:35.107174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.194:36953 (local address 127.6.246.195:47599)
0925 22:36:35.107907 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:35.107919 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:35.107982 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:35.109649 (+ 1667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:35.109652 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:35.109669 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:36:35.109775 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:35.109782 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:35.110840 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:35.110843 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:35.112485 (+ 1642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:35.112495 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:35.113217 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:35.113221 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:35.113325 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:35.113358 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 22:36:35.113466 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 22:36:35.113531 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"thread_start_us":56,"threads_started":1}
I20260925 22:36:35.113691 15178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:35.107572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.195:47599 (local address 127.6.246.194:36953)
0925 22:36:35.108040 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:35.108044 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:35.108056 (+ 12us) server_negotiation.cc:407] Connection header received
0925 22:36:35.108097 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:35.108100 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:35.108140 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:35.108213 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:36:35.109905 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:35.110686 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:35.112630 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:35.113074 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:35.113197 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:35.113248 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:35.113494 (+ 246us) server_negotiation.cc:299] Negotiation successful
0925 22:36:35.113578 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395}
I20260925 22:36:35.114065 14966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40d4c4a6ffe24789a5b9715ecc1960a8" candidate_uuid: "c99f2a7a833f49c68022c8a09e09c9ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "063799ab6b434788be1447722615f67d" is_pre_election: true
I20260925 22:36:35.114202 14966 raft_consensus.cc:2454] T 40d4c4a6ffe24789a5b9715ecc1960a8 P 063799ab6b434788be1447722615f67d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c99f2a7a833f49c68022c8a09e09c9ef in term 0.
I20260925 22:36:35.114583 15187 raft_consensus.cc:504] T 40d4c4a6ffe24789a5b9715ecc1960a8 P c99f2a7a833f49c68022c8a09e09c9ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35ce5c2bbc9406e86e15892c07a97fd" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 38857 } } peers { permanent_uuid: "063799ab6b434788be1447722615f67d" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 36953 } } peers { permanent_uuid: "c99f2a7a833f49c68022c8a09e09c9ef" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 44587 } }
I20260925 22:36:35.114931 15187 leader_election.cc:288] T 40d4c4a6ffe24789a5b9715ecc1960a8 P c99f2a7a833f49c68022c8a09e09c9ef [CANDIDATE]: Term 1 election: Requested vote from peers e35ce5c2bbc9406e86e15892c07a97fd (127.6.246.193:38857), 063799ab6b434788be1447722615f67d (127.6.246.194:36953)
I20260925 22:36:35.115211 14827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40d4c4a6ffe24789a5b9715ecc1960a8" candidate_uuid: "c99f2a7a833f49c68022c8a09e09c9ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e35ce5c2bbc9406e86e15892c07a97fd"
I20260925 22:36:35.115258 14966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40d4c4a6ffe24789a5b9715ecc1960a8" candidate_uuid: "c99f2a7a833f49c68022c8a09e09c9ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "063799ab6b434788be1447722615f67d"
I20260925 22:36:35.115324 14827 raft_consensus.cc:3037] T 40d4c4a6ffe24789a5b9715ecc1960a8 P e35ce5c2bbc9406e86e15892c07a97fd [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:36:35.115376 14966 raft_consensus.cc:3037] T 40d4c4a6ffe24789a5b9715ecc1960a8 P 063799ab6b434788be1447722615f67d [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:36:35.116173 14827 raft_consensus.cc:2454] T 40d4c4a6ffe24789a5b9715ecc1960a8 P e35ce5c2bbc9406e86e15892c07a97fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c99f2a7a833f49c68022c8a09e09c9ef in term 1.
I20260925 22:36:35.116523 15038 leader_election.cc:302] T 40d4c4a6ffe24789a5b9715ecc1960a8 P c99f2a7a833f49c68022c8a09e09c9ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c99f2a7a833f49c68022c8a09e09c9ef, e35ce5c2bbc9406e86e15892c07a97fd; no voters:
I20260925 22:36:35.116537 14966 raft_consensus.cc:2454] T 40d4c4a6ffe24789a5b9715ecc1960a8 P 063799ab6b434788be1447722615f67d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c99f2a7a833f49c68022c8a09e09c9ef in term 1.
I20260925 22:36:35.116927 15187 raft_consensus.cc:2781] T 40d4c4a6ffe24789a5b9715ecc1960a8 P c99f2a7a833f49c68022c8a09e09c9ef [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:36:35.117143 15187 raft_consensus.cc:686] T 40d4c4a6ffe24789a5b9715ecc1960a8 P c99f2a7a833f49c68022c8a09e09c9ef [term 1 LEADER]: Becoming Leader. State: Replica: c99f2a7a833f49c68022c8a09e09c9ef, State: Running, Role: LEADER
I20260925 22:36:35.117450 15187 consensus_queue.cc:237] T 40d4c4a6ffe24789a5b9715ecc1960a8 P c99f2a7a833f49c68022c8a09e09c9ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35ce5c2bbc9406e86e15892c07a97fd" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 38857 } } peers { permanent_uuid: "063799ab6b434788be1447722615f67d" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 36953 } } peers { permanent_uuid: "c99f2a7a833f49c68022c8a09e09c9ef" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 44587 } }
I20260925 22:36:35.119640 14669 catalog_manager.cc:5495] T 40d4c4a6ffe24789a5b9715ecc1960a8 P c99f2a7a833f49c68022c8a09e09c9ef reported cstate change: term changed from 0 to 1, leader changed from <none> to c99f2a7a833f49c68022c8a09e09c9ef (127.6.246.195). New cstate: current_term: 1 leader_uuid: "c99f2a7a833f49c68022c8a09e09c9ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35ce5c2bbc9406e86e15892c07a97fd" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 38857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "063799ab6b434788be1447722615f67d" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 36953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c99f2a7a833f49c68022c8a09e09c9ef" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 44587 } health_report { overall_health: HEALTHY } } }
I20260925 22:36:35.134426 15249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:35.130686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54584 (local address 127.6.246.195:44587)
0925 22:36:35.130797 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:35.130801 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:35.130827 (+ 26us) server_negotiation.cc:407] Connection header received
0925 22:36:35.130967 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:35.130969 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:35.131017 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:35.131135 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:36:35.131365 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:35.131923 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:35.133002 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:35.133223 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:35.133349 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:35.133601 (+ 252us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:36:35.134044 (+ 443us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:36:35.134167 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:35.134224 (+ 57us) server_negotiation.cc:299] Negotiation successful
0925 22:36:35.134274 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 22:36:35.144337 15178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:35.140962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.195:51313 (local address 127.6.246.194:36953)
0925 22:36:35.141097 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:35.141101 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:35.141191 (+ 90us) server_negotiation.cc:407] Connection header received
0925 22:36:35.141294 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:35.141296 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:35.141335 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:35.141406 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:36:35.141652 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:35.142115 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:35.143344 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:35.143575 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:35.143666 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:35.143956 (+ 290us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:36:35.144017 (+ 61us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:36:35.144073 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:35.144190 (+ 117us) server_negotiation.cc:299] Negotiation successful
0925 22:36:35.144232 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 22:36:35.144423 15266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:35.140813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.194:36953 (local address 127.6.246.195:51313)
0925 22:36:35.141108 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:35.141123 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:35.141218 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:35.141433 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:35.141436 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:35.141443 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 22:36:35.141554 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:35.141560 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:35.142219 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:35.142222 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:35.143212 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:35.143222 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:35.143691 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:35.143695 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:35.143786 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:35.143832 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 22:36:35.144103 (+ 271us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 22:36:35.144109 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 22:36:35.144173 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 22:36:35.144247 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":116,"threads_started":1}
I20260925 22:36:35.150085 14827 raft_consensus.cc:1261] T 40d4c4a6ffe24789a5b9715ecc1960a8 P e35ce5c2bbc9406e86e15892c07a97fd [term 1 FOLLOWER]: Refusing update from remote peer c99f2a7a833f49c68022c8a09e09c9ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:36:35.150146 14966 raft_consensus.cc:1261] T 40d4c4a6ffe24789a5b9715ecc1960a8 P 063799ab6b434788be1447722615f67d [term 1 FOLLOWER]: Refusing update from remote peer c99f2a7a833f49c68022c8a09e09c9ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:36:35.150559 15262 consensus_queue.cc:1035] T 40d4c4a6ffe24789a5b9715ecc1960a8 P c99f2a7a833f49c68022c8a09e09c9ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "e35ce5c2bbc9406e86e15892c07a97fd" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 38857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:36:35.150741 15187 consensus_queue.cc:1035] T 40d4c4a6ffe24789a5b9715ecc1960a8 P c99f2a7a833f49c68022c8a09e09c9ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "063799ab6b434788be1447722615f67d" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 36953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:36:35.173271 15228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:35.169233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.195:44587 (local address 127.6.246.194:53339)
0925 22:36:35.169402 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:35.169414 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:35.169477 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:35.169797 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:35.169799 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:35.169805 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 22:36:35.169903 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:35.169909 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:35.170933 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:35.170936 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:35.171890 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:35.171900 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:35.172563 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:35.172567 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:35.172660 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:35.172690 (+ 30us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 22:36:35.172983 (+ 293us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 22:36:35.172989 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 22:36:35.173047 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 22:36:35.173135 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260925 22:36:35.173636 15249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:35.169271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.194:53339 (local address 127.6.246.195:44587)
0925 22:36:35.169435 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:35.169439 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:35.169454 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:36:35.169581 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:35.169584 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:35.169631 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:35.169696 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:36:35.170015 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:35.170803 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:35.172018 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:35.172369 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:35.172483 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:35.172810 (+ 327us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:36:35.172890 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:36:35.172949 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:35.173463 (+ 514us) server_negotiation.cc:299] Negotiation successful
0925 22:36:35.173522 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260925 22:36:35.197638 14668 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:36:35.197803 14668 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:36:35.199663 14668 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 302768cab249462291567a64833dd1a1: Sending DeleteTablet for 3 replicas of tablet 40d4c4a6ffe24789a5b9715ecc1960a8
I20260925 22:36:35.200364 14807 tablet_service.cc:1514] Processing DeleteTablet for tablet 40d4c4a6ffe24789a5b9715ecc1960a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:36:35 UTC) from {username='oryx', principal='oryx/127.6.246.254@KRBTEST.COM'} at 127.0.0.1:58542
I20260925 22:36:35.200430 14946 tablet_service.cc:1514] Processing DeleteTablet for tablet 40d4c4a6ffe24789a5b9715ecc1960a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:36:35 UTC) from {username='oryx', principal='oryx/127.6.246.254@KRBTEST.COM'} at 127.0.0.1:45972
I20260925 22:36:35.200562 15087 tablet_service.cc:1514] Processing DeleteTablet for tablet 40d4c4a6ffe24789a5b9715ecc1960a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:36:35 UTC) from {username='oryx', principal='oryx/127.6.246.254@KRBTEST.COM'} at 127.0.0.1:54576
I20260925 22:36:35.200600 15204 tablet_replica.cc:330] T 40d4c4a6ffe24789a5b9715ecc1960a8 P 063799ab6b434788be1447722615f67d: stopping tablet replica
I20260925 22:36:35.200700 15204 raft_consensus.cc:2229] T 40d4c4a6ffe24789a5b9715ecc1960a8 P 063799ab6b434788be1447722615f67d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:36:35.200716 15205 tablet_replica.cc:330] T 40d4c4a6ffe24789a5b9715ecc1960a8 P e35ce5c2bbc9406e86e15892c07a97fd: stopping tablet replica
I20260925 22:36:35.200804 15204 raft_consensus.cc:2258] T 40d4c4a6ffe24789a5b9715ecc1960a8 P 063799ab6b434788be1447722615f67d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:36:35.200831 15205 raft_consensus.cc:2229] T 40d4c4a6ffe24789a5b9715ecc1960a8 P e35ce5c2bbc9406e86e15892c07a97fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:36:35.200922 15205 raft_consensus.cc:2258] T 40d4c4a6ffe24789a5b9715ecc1960a8 P e35ce5c2bbc9406e86e15892c07a97fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:36:35.201066 15206 tablet_replica.cc:330] T 40d4c4a6ffe24789a5b9715ecc1960a8 P c99f2a7a833f49c68022c8a09e09c9ef: stopping tablet replica
I20260925 22:36:35.201181 15206 raft_consensus.cc:2229] T 40d4c4a6ffe24789a5b9715ecc1960a8 P c99f2a7a833f49c68022c8a09e09c9ef [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:36:35.201452 15206 raft_consensus.cc:2258] T 40d4c4a6ffe24789a5b9715ecc1960a8 P c99f2a7a833f49c68022c8a09e09c9ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:36:35.201581 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 14742
I20260925 22:36:35.201634 15204 ts_tablet_manager.cc:1905] T 40d4c4a6ffe24789a5b9715ecc1960a8 P 063799ab6b434788be1447722615f67d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:36:35.203586 15206 ts_tablet_manager.cc:1905] T 40d4c4a6ffe24789a5b9715ecc1960a8 P c99f2a7a833f49c68022c8a09e09c9ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:36:35.204207 15204 ts_tablet_manager.cc:1918] T 40d4c4a6ffe24789a5b9715ecc1960a8 P 063799ab6b434788be1447722615f67d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 22:36:35.204301 15204 log.cc:1196] T 40d4c4a6ffe24789a5b9715ecc1960a8 P 063799ab6b434788be1447722615f67d: Deleting WAL directory at /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/minicluster-data/ts-1/wal/wals/40d4c4a6ffe24789a5b9715ecc1960a8
I20260925 22:36:35.204607 15204 ts_tablet_manager.cc:1939] T 40d4c4a6ffe24789a5b9715ecc1960a8 P 063799ab6b434788be1447722615f67d: Deleting consensus metadata
I20260925 22:36:35.205236 14655 catalog_manager.cc:4841] TS 063799ab6b434788be1447722615f67d (127.6.246.194:36953): tablet 40d4c4a6ffe24789a5b9715ecc1960a8 (table test-table [id=50319928ebb14f1aab49eeef8532bd9e]) successfully deleted
I20260925 22:36:35.206552 15206 ts_tablet_manager.cc:1918] T 40d4c4a6ffe24789a5b9715ecc1960a8 P c99f2a7a833f49c68022c8a09e09c9ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 22:36:35.206633 15206 log.cc:1196] T 40d4c4a6ffe24789a5b9715ecc1960a8 P c99f2a7a833f49c68022c8a09e09c9ef: Deleting WAL directory at /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790375752797504-7131-0/minicluster-data/ts-2/wal/wals/40d4c4a6ffe24789a5b9715ecc1960a8
I20260925 22:36:35.206892 15206 ts_tablet_manager.cc:1939] T 40d4c4a6ffe24789a5b9715ecc1960a8 P c99f2a7a833f49c68022c8a09e09c9ef: Deleting consensus metadata
I20260925 22:36:35.207553 14654 catalog_manager.cc:4841] TS c99f2a7a833f49c68022c8a09e09c9ef (127.6.246.195:44587): tablet 40d4c4a6ffe24789a5b9715ecc1960a8 (table test-table [id=50319928ebb14f1aab49eeef8532bd9e]) successfully deleted
W20260925 22:36:35.210060 14654 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.246.193:38857 (error 108)
I20260925 22:36:35.210192 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 14882
I20260925 22:36:35.216842 15272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:35.211120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.193:38857 (local address 127.0.0.1:58570)
0925 22:36:35.216693 (+ 5573us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:35.216757 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.193:38857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5503,"thread_start_us":53,"threads_started":1}
W20260925 22:36:35.216975 14654 catalog_manager.cc:4584] TS e35ce5c2bbc9406e86e15892c07a97fd (127.6.246.193:38857): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 40d4c4a6ffe24789a5b9715ecc1960a8: Network error: Client connection negotiation failed: client connection to 127.6.246.193:38857: connect: Connection refused (error 111)
I20260925 22:36:35.219872 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 15022
I20260925 22:36:35.225432 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 14638
2026-09-25T22:36:35Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3093 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790375752797504-7131-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:36:35 dist-test-slave-wjdr krb5kdc[15275](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:36:35 dist-test-slave-wjdr krb5kdc[15275](info): set up 2 sockets
Sep 25 22:36:35 dist-test-slave-wjdr krb5kdc[15275](info): commencing operation
krb5kdc: starting...
W20260925 22:36:36.265182 7131 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Sep 25 22:36:36 dist-test-slave-wjdr krb5kdc[15275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:36:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:36:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.246.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab.
Entry for principal kudu/127.6.246.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab.
I20260925 22:36:36.389559 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790375752797504-7131-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790375752797504-7131-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790375752797504-7131-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:37501
--webserver_interface=127.6.246.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab
--principal=kudu/127.6.246.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:45803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.246.254:37501
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:36:36.498425 15288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:36:36.498697 15288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:36:36.498739 15288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:36:36.502736 15288 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:36:36.502794 15288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:36:36.502812 15288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:36:36.502828 15288 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:36:36.502898 15288 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:36:36.508119 15288 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:45803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790375752797504-7131-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790375752797504-7131-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.246.254:37501
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:37501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.246.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15288
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790375752797504-7131-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:36:36.509114 15288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:36:36.510111 15288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:36:36.518612 15294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:36.518592 15296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:36.518647 15293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:36.519088 15288 server_base.cc:1034] running on GCE node
I20260925 22:36:36.519598 15288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:36:36.520689 15288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:36:36.521879 15288 hybrid_clock.cc:648] HybridClock initialized: now 1790375796521860 us; error 30 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-25T22:36:36Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1300 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:36:36 dist-test-slave-wjdr krb5kdc[15301](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:36:36 dist-test-slave-wjdr krb5kdc[15301](info): set up 2 sockets
Sep 25 22:36:36 dist-test-slave-wjdr krb5kdc[15301](info): commencing operation
krb5kdc: starting...
W20260925 22:36:37.581476 7131 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Sep 25 22:36:37 dist-test-slave-wjdr krb5kdc[15301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375797, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:36:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:36:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.246.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab.
Entry for principal kudu/127.6.246.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab.
I20260925 22:36:37.699442 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:34777
--webserver_interface=127.6.246.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab
--principal=kudu/127.6.246.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:46577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.246.254:34777
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:36:37.808702 15314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:36:37.808957 15314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:36:37.808998 15314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:36:37.812913 15314 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:36:37.812961 15314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:36:37.812980 15314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:36:37.812995 15314 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:36:37.813010 15314 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:36:37.818303 15314 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:46577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.246.254:34777
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab
--principal=kudu/127.6.246.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:34777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.246.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15314
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:36:37.819252 15314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:36:37.820194 15314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:36:37.828413 15319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:37.828449 15314 server_base.cc:1034] running on GCE node
W20260925 22:36:37.828488 15322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:37.828413 15320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:37.829265 15314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:36:37.830257 15314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:36:37.831420 15314 hybrid_clock.cc:648] HybridClock initialized: now 1790375797831393 us; error 42 us; skew 500 ppm
Sep 25 22:36:37 dist-test-slave-wjdr krb5kdc[15301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375797, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:36:37.834183 15314 init.cc:377] Logged in from keytab as kudu/127.6.246.254@KRBTEST.COM (short username kudu)
I20260925 22:36:37.835083 15314 webserver.cc:460] Webserver started at http://127.6.246.254:33301/ using document root <none> and password file <none>
I20260925 22:36:37.835632 15314 fs_manager.cc:359] Metadata directory not provided
I20260925 22:36:37.835718 15314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:36:37.835950 15314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:36:37.838387 15314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/master-0/data/instance:
uuid: "7e697e1ddfb7436aa6aeb550f3015556"
format_stamp: "Formatted at 2026-09-25 22:36:37 on dist-test-slave-wjdr"
server_key: "457d6e014b750950044f99e06e7ffa77"
server_key_iv: "120dbf81f02ad4cdcec7dc5808a0308b"
server_key_version: "encryptionkey@0"
I20260925 22:36:37.838948 15314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/master-0/wal/instance:
uuid: "7e697e1ddfb7436aa6aeb550f3015556"
format_stamp: "Formatted at 2026-09-25 22:36:37 on dist-test-slave-wjdr"
server_key: "457d6e014b750950044f99e06e7ffa77"
server_key_iv: "120dbf81f02ad4cdcec7dc5808a0308b"
server_key_version: "encryptionkey@0"
I20260925 22:36:37.842572 15314 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20260925 22:36:37.844972 15329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:37.846199 15314 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:36:37.846352 15314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/master-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/master-0/wal
uuid: "7e697e1ddfb7436aa6aeb550f3015556"
format_stamp: "Formatted at 2026-09-25 22:36:37 on dist-test-slave-wjdr"
server_key: "457d6e014b750950044f99e06e7ffa77"
server_key_iv: "120dbf81f02ad4cdcec7dc5808a0308b"
server_key_version: "encryptionkey@0"
I20260925 22:36:37.846473 15314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:36:37.866724 15314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:36:37.869997 15314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:36:37.870203 15314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:36:37.877396 15314 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.254:34777
I20260925 22:36:37.877417 15381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.254:34777 every 8 connection(s)
I20260925 22:36:37.878328 15314 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
I20260925 22:36:37.880961 15382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:36:37.885413 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 15314
I20260925 22:36:37.885551 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/master-0/wal/instance
I20260925 22:36:37.885867 7131 external_mini_cluster.cc:1428] Setting key 6f57442b615f237a2e65b3ca4455d05d
I20260925 22:36:37.886698 15382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e697e1ddfb7436aa6aeb550f3015556: Bootstrap starting.
I20260925 22:36:37.889113 15382 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e697e1ddfb7436aa6aeb550f3015556: Neither blocks nor log segments found. Creating new log.
I20260925 22:36:37.889858 15382 log.cc:824] T 00000000000000000000000000000000 P 7e697e1ddfb7436aa6aeb550f3015556: Log is configured to *not* fsync() on all Append() calls
I20260925 22:36:37.891618 15382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e697e1ddfb7436aa6aeb550f3015556: No bootstrap required, opened a new log
Sep 25 22:36:37 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375797, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
I20260925 22:36:37.895638 15382 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e697e1ddfb7436aa6aeb550f3015556 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e697e1ddfb7436aa6aeb550f3015556" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 34777 } }
I20260925 22:36:37.895947 15382 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e697e1ddfb7436aa6aeb550f3015556 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:36:37.896036 15382 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e697e1ddfb7436aa6aeb550f3015556 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e697e1ddfb7436aa6aeb550f3015556, State: Initialized, Role: FOLLOWER
I20260925 22:36:37.896586 15382 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e697e1ddfb7436aa6aeb550f3015556 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e697e1ddfb7436aa6aeb550f3015556" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 34777 } }
I20260925 22:36:37.896719 15382 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e697e1ddfb7436aa6aeb550f3015556 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:36:37.896813 15382 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e697e1ddfb7436aa6aeb550f3015556 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:36:37.896915 15382 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e697e1ddfb7436aa6aeb550f3015556 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:36:37.897794 15382 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e697e1ddfb7436aa6aeb550f3015556 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e697e1ddfb7436aa6aeb550f3015556" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 34777 } }
I20260925 22:36:37.898136 15382 leader_election.cc:302] T 00000000000000000000000000000000 P 7e697e1ddfb7436aa6aeb550f3015556 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e697e1ddfb7436aa6aeb550f3015556; no voters:
I20260925 22:36:37.898480 15382 leader_election.cc:288] T 00000000000000000000000000000000 P 7e697e1ddfb7436aa6aeb550f3015556 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:36:37.898548 15387 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e697e1ddfb7436aa6aeb550f3015556 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:36:37.898741 15387 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e697e1ddfb7436aa6aeb550f3015556 [term 1 LEADER]: Becoming Leader. State: Replica: 7e697e1ddfb7436aa6aeb550f3015556, State: Running, Role: LEADER
I20260925 22:36:37.899085 15387 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e697e1ddfb7436aa6aeb550f3015556 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e697e1ddfb7436aa6aeb550f3015556" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 34777 } }
I20260925 22:36:37.899747 15382 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e697e1ddfb7436aa6aeb550f3015556 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:36:37.900671 15388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e697e1ddfb7436aa6aeb550f3015556 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e697e1ddfb7436aa6aeb550f3015556" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e697e1ddfb7436aa6aeb550f3015556" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 34777 } } }
I20260925 22:36:37.900811 15388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e697e1ddfb7436aa6aeb550f3015556 [sys.catalog]: This master's current role is: LEADER
I20260925 22:36:37.901010 15385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:37.887284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42870 (local address 127.6.246.254:34777)
0925 22:36:37.887712 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:37.887726 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:37.887760 (+ 34us) server_negotiation.cc:407] Connection header received
0925 22:36:37.888398 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:37.888414 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:37.888808 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:37.889138 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:37.889983 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:37.890984 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:37.891880 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:37.892330 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:37.892528 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:37.895335 (+ 2807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:37.895340 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:37.895355 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:37.895382 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:37.897361 (+ 1979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:37.897790 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:37.897796 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:37.897802 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:37.897982 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:37.898258 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:37.898263 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:37.898265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:37.898804 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:37.899601 (+ 797us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:37.899717 (+ 116us) server_negotiation.cc:299] Negotiation successful
0925 22:36:37.900016 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":132,"threads_started":1}
I20260925 22:36:37.901054 15388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e697e1ddfb7436aa6aeb550f3015556 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e697e1ddfb7436aa6aeb550f3015556. Latest consensus state: current_term: 1 leader_uuid: "7e697e1ddfb7436aa6aeb550f3015556" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e697e1ddfb7436aa6aeb550f3015556" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 34777 } } }
I20260925 22:36:37.901144 15388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e697e1ddfb7436aa6aeb550f3015556 [sys.catalog]: This master's current role is: LEADER
W20260925 22:36:37.905030 15403 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7e697e1ddfb7436aa6aeb550f3015556: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:36:37.905121 15403 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:36:37.905205 15396 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:36:37.905880 15396 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:36:37.910878 15396 catalog_manager.cc:1347] Generated new cluster ID: 633b4857a853448d9ecdd04d5310fde6
I20260925 22:36:37.910945 15396 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:36:37.931087 15396 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:36:37.932350 15396 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:36:37.940807 15396 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e697e1ddfb7436aa6aeb550f3015556: Generated new TSK 0
I20260925 22:36:37.941490 15396 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.246.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.193.keytab.
Entry for principal kudu/127.6.246.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.193.keytab.
I20260925 22:36:37.975992 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.193:0
--local_ip_for_outbound_sockets=127.6.246.193
--webserver_interface=127.6.246.193
--webserver_port=0
--tserver_master_addrs=127.6.246.254:34777
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.193.keytab
--principal=kudu/127.6.246.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:46577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:36:38.102867 15408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:36:38.103123 15408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:36:38.103165 15408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:36:38.107090 15408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:36:38.107146 15408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:36:38.107203 15408 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.193
I20260925 22:36:38.112421 15408 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:46577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.193.keytab
--principal=kudu/127.6.246.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.246.193
--webserver_port=0
--tserver_master_addrs=127.6.246.254:34777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15408
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.193
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:36:38.113430 15408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:36:38.114457 15408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:36:38.123974 15413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:38.123965 15414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:38.123965 15416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:38.124389 15408 server_base.cc:1034] running on GCE node
I20260925 22:36:38.124802 15408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:36:38.125433 15408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:36:38.126618 15408 hybrid_clock.cc:648] HybridClock initialized: now 1790375798126608 us; error 27 us; skew 500 ppm
Sep 25 22:36:38 dist-test-slave-wjdr krb5kdc[15301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375798, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:36:38.129467 15408 init.cc:377] Logged in from keytab as kudu/127.6.246.193@KRBTEST.COM (short username kudu)
I20260925 22:36:38.130244 15408 webserver.cc:460] Webserver started at http://127.6.246.193:38581/ using document root <none> and password file <none>
I20260925 22:36:38.130764 15408 fs_manager.cc:359] Metadata directory not provided
I20260925 22:36:38.130821 15408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:36:38.130986 15408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:36:38.133096 15408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-0/data/instance:
uuid: "a7b2ec64e6a445b49615c39f03d5f0b6"
format_stamp: "Formatted at 2026-09-25 22:36:38 on dist-test-slave-wjdr"
server_key: "0d574358439dd712463f0e2d1845f506"
server_key_iv: "365edb238ff469dfd22737c3035b7f16"
server_key_version: "encryptionkey@0"
I20260925 22:36:38.133612 15408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-0/wal/instance:
uuid: "a7b2ec64e6a445b49615c39f03d5f0b6"
format_stamp: "Formatted at 2026-09-25 22:36:38 on dist-test-slave-wjdr"
server_key: "0d574358439dd712463f0e2d1845f506"
server_key_iv: "365edb238ff469dfd22737c3035b7f16"
server_key_version: "encryptionkey@0"
I20260925 22:36:38.137351 15408 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:36:38.139700 15423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:38.141000 15408 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:36:38.141127 15408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-0/wal
uuid: "a7b2ec64e6a445b49615c39f03d5f0b6"
format_stamp: "Formatted at 2026-09-25 22:36:38 on dist-test-slave-wjdr"
server_key: "0d574358439dd712463f0e2d1845f506"
server_key_iv: "365edb238ff469dfd22737c3035b7f16"
server_key_version: "encryptionkey@0"
I20260925 22:36:38.141228 15408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:36:38.164908 15408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:36:38.168187 15408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:36:38.168395 15408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:36:38.169098 15408 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:36:38.169894 15408 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:36:38.169963 15408 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:38.170035 15408 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:36:38.170087 15408 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:38.179582 15408 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.193:38715
I20260925 22:36:38.179603 15536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.193:38715 every 8 connection(s)
I20260925 22:36:38.180517 15408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
I20260925 22:36:38.183220 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 15408
I20260925 22:36:38.183374 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-0/wal/instance
I20260925 22:36:38.183722 7131 external_mini_cluster.cc:1428] Setting key 277d697269b7fd386c152407326fdf2c
Sep 25 22:36:38 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375798, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.193@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.246.194@KRBTEST.COM; defaulting to no policy
I20260925 22:36:38.195421 15385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:38.182311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.193:44879 (local address 127.6.246.254:34777)
0925 22:36:38.182454 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:38.182458 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:38.183650 (+ 1192us) server_negotiation.cc:407] Connection header received
0925 22:36:38.184776 (+ 1126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:38.184779 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:38.184823 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:38.184920 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:38.186483 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:38.187144 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:38.188265 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:38.188706 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:38.188830 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:38.191405 (+ 2575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:38.191410 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:38.191413 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:38.191444 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:38.193302 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:38.193948 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:38.193952 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:38.193954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:38.194047 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:38.194373 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:38.194375 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:38.194376 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:38.194546 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:38.194669 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:38.195196 (+ 527us) server_negotiation.cc:299] Negotiation successful
0925 22:36:38.195298 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 22:36:38.196321 15539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:38.182659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:34777 (local address 127.6.246.193:44879)
0925 22:36:38.183475 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:38.183535 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:38.184582 (+ 1047us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:38.185135 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:38.185142 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:38.185639 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:38.186329 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:38.186350 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:38.187274 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:38.187277 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:38.188137 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:38.188146 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:38.188875 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:38.188878 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:38.189036 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:38.189763 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:38.189780 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:38.191234 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:36:38.193458 (+ 2224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:38.193463 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:38.193476 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:38.193841 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:38.194141 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:38.194143 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:38.194144 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:38.194290 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:38.194717 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:36:38.194727 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:36:38.194985 (+ 258us) client_negotiation.cc:769] Sending connection context
0925 22:36:38.195149 (+ 164us) client_negotiation.cc:240] Negotiation successful
0925 22:36:38.195409 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":113,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.194@KRBTEST.COM" created.
I20260925 22:36:38.197649 15537 heartbeater.cc:347] Connected to a master server at 127.6.246.254:34777
I20260925 22:36:38.197896 15537 heartbeater.cc:464] Registering TS with master...
I20260925 22:36:38.198379 15537 heartbeater.cc:511] Master 127.6.246.254:34777 requested a full tablet report, sending...
I20260925 22:36:38.199896 15346 ts_manager.cc:194] Registered new tserver with Master: a7b2ec64e6a445b49615c39f03d5f0b6 (127.6.246.193:38715)
I20260925 22:36:38.201207 15346 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.246.193@KRBTEST.COM'} at 127.6.246.193:44879
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.194.keytab.
Entry for principal kudu/127.6.246.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.194.keytab.
I20260925 22:36:38.212163 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.194:0
--local_ip_for_outbound_sockets=127.6.246.194
--webserver_interface=127.6.246.194
--webserver_port=0
--tserver_master_addrs=127.6.246.254:34777
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.194.keytab
--principal=kudu/127.6.246.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:46577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:36:38.320392 15542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:36:38.320710 15542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:36:38.320778 15542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:36:38.324753 15542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:36:38.324841 15542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:36:38.324939 15542 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.194
I20260925 22:36:38.330150 15542 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:46577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.194.keytab
--principal=kudu/127.6.246.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.246.194
--webserver_port=0
--tserver_master_addrs=127.6.246.254:34777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15542
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.194
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:36:38.331164 15542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:36:38.332154 15542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:36:38.341472 15550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:38.341553 15548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:38.341495 15547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:38.341832 15542 server_base.cc:1034] running on GCE node
I20260925 22:36:38.342165 15542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:36:38.342742 15542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:36:38.343923 15542 hybrid_clock.cc:648] HybridClock initialized: now 1790375798343912 us; error 24 us; skew 500 ppm
Sep 25 22:36:38 dist-test-slave-wjdr krb5kdc[15301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375798, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:36:38.346580 15542 init.cc:377] Logged in from keytab as kudu/127.6.246.194@KRBTEST.COM (short username kudu)
I20260925 22:36:38.347437 15542 webserver.cc:460] Webserver started at http://127.6.246.194:35893/ using document root <none> and password file <none>
I20260925 22:36:38.347970 15542 fs_manager.cc:359] Metadata directory not provided
I20260925 22:36:38.348057 15542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:36:38.348271 15542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:36:38.350437 15542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-1/data/instance:
uuid: "006ee5a095c443949c0baa2ffd9682d3"
format_stamp: "Formatted at 2026-09-25 22:36:38 on dist-test-slave-wjdr"
server_key: "69ee9e8688707525b1ee332373289d84"
server_key_iv: "52a97f6077625000aaac38b31c2d88e9"
server_key_version: "encryptionkey@0"
I20260925 22:36:38.351003 15542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-1/wal/instance:
uuid: "006ee5a095c443949c0baa2ffd9682d3"
format_stamp: "Formatted at 2026-09-25 22:36:38 on dist-test-slave-wjdr"
server_key: "69ee9e8688707525b1ee332373289d84"
server_key_iv: "52a97f6077625000aaac38b31c2d88e9"
server_key_version: "encryptionkey@0"
I20260925 22:36:38.354588 15542 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 22:36:38.356930 15557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:38.358124 15542 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:36:38.358251 15542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-1/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-1/wal
uuid: "006ee5a095c443949c0baa2ffd9682d3"
format_stamp: "Formatted at 2026-09-25 22:36:38 on dist-test-slave-wjdr"
server_key: "69ee9e8688707525b1ee332373289d84"
server_key_iv: "52a97f6077625000aaac38b31c2d88e9"
server_key_version: "encryptionkey@0"
I20260925 22:36:38.358371 15542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:36:38.387068 15542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:36:38.390271 15542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:36:38.390491 15542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:36:38.391094 15542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:36:38.391908 15542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:36:38.391978 15542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:38.392071 15542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:36:38.392118 15542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:38.401269 15542 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.194:38425
I20260925 22:36:38.401332 15670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.194:38425 every 8 connection(s)
I20260925 22:36:38.402202 15542 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
I20260925 22:36:38.408417 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 15542
I20260925 22:36:38.408571 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-1/wal/instance
I20260925 22:36:38.408869 7131 external_mini_cluster.cc:1428] Setting key 43c4b4aca25a5f0f9bc419095902b7ae
Sep 25 22:36:38 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375798, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.194@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
I20260925 22:36:38.416352 15385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:38.403961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.194:58983 (local address 127.6.246.254:34777)
0925 22:36:38.404087 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:38.404093 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:38.405182 (+ 1089us) server_negotiation.cc:407] Connection header received
0925 22:36:38.406048 (+ 866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:38.406051 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:38.406107 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:38.406186 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:38.407270 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:38.407802 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:38.409347 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:38.409596 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:38.409715 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:38.412323 (+ 2608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:38.412329 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:38.412333 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:38.412364 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:38.414075 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:38.414662 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:38.414668 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:38.414671 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:38.414768 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:38.415091 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:38.415097 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:38.415102 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:38.415370 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:38.415482 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:38.415960 (+ 478us) server_negotiation.cc:299] Negotiation successful
0925 22:36:38.416114 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 22:36:38.417039 15673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:38.404345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:34777 (local address 127.6.246.194:58983)
0925 22:36:38.405028 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:38.405067 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:38.405854 (+ 787us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:38.406340 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:38.406346 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:38.406683 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:38.407131 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:38.407143 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:38.407935 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:38.407938 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:38.409224 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:38.409240 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:38.409745 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:38.409749 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:38.409917 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:38.410520 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:38.410536 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:38.412158 (+ 1622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:36:38.414194 (+ 2036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:38.414199 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:38.414210 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:38.414552 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:38.414864 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:38.414866 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:38.414867 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:38.414997 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:38.415484 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:36:38.415488 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:36:38.415716 (+ 228us) client_negotiation.cc:769] Sending connection context
0925 22:36:38.415869 (+ 153us) client_negotiation.cc:240] Negotiation successful
0925 22:36:38.416131 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":163,"threads_started":1}
I20260925 22:36:38.418193 15671 heartbeater.cc:347] Connected to a master server at 127.6.246.254:34777
I20260925 22:36:38.418422 15671 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.6.246.195@KRBTEST.COM; defaulting to no policy
I20260925 22:36:38.418936 15671 heartbeater.cc:511] Master 127.6.246.254:34777 requested a full tablet report, sending...
I20260925 22:36:38.420004 15346 ts_manager.cc:194] Registered new tserver with Master: 006ee5a095c443949c0baa2ffd9682d3 (127.6.246.194:38425)
I20260925 22:36:38.420637 15346 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.246.194@KRBTEST.COM'} at 127.6.246.194:58983
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.195.keytab.
Entry for principal kudu/127.6.246.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.195.keytab.
I20260925 22:36:38.437815 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.195:0
--local_ip_for_outbound_sockets=127.6.246.195
--webserver_interface=127.6.246.195
--webserver_port=0
--tserver_master_addrs=127.6.246.254:34777
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.195.keytab
--principal=kudu/127.6.246.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:46577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:36:38.547120 15676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:36:38.547380 15676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:36:38.547422 15676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:36:38.551496 15676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:36:38.551560 15676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:36:38.551620 15676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.195
I20260925 22:36:38.556865 15676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:46577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.195.keytab
--principal=kudu/127.6.246.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.246.195
--webserver_port=0
--tserver_master_addrs=127.6.246.254:34777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15676
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.195
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:36:38.557826 15676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:36:38.558792 15676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:36:38.568269 15682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:38.568342 15684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:38.568378 15676 server_base.cc:1034] running on GCE node
W20260925 22:36:38.568660 15681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:38.569046 15676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:36:38.569667 15676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:36:38.570859 15676 hybrid_clock.cc:648] HybridClock initialized: now 1790375798570831 us; error 39 us; skew 500 ppm
Sep 25 22:36:38 dist-test-slave-wjdr krb5kdc[15301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375798, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:36:38.573585 15676 init.cc:377] Logged in from keytab as kudu/127.6.246.195@KRBTEST.COM (short username kudu)
I20260925 22:36:38.574469 15676 webserver.cc:460] Webserver started at http://127.6.246.195:41243/ using document root <none> and password file <none>
I20260925 22:36:38.575009 15676 fs_manager.cc:359] Metadata directory not provided
I20260925 22:36:38.575096 15676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:36:38.575315 15676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:36:38.577345 15676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-2/data/instance:
uuid: "027d99d1e6ec4a54b31ddb654d0b78a6"
format_stamp: "Formatted at 2026-09-25 22:36:38 on dist-test-slave-wjdr"
server_key: "bf265a83aea2688af8ad088283e21a29"
server_key_iv: "1551b264527c9805761425ccf135ec1a"
server_key_version: "encryptionkey@0"
I20260925 22:36:38.577899 15676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-2/wal/instance:
uuid: "027d99d1e6ec4a54b31ddb654d0b78a6"
format_stamp: "Formatted at 2026-09-25 22:36:38 on dist-test-slave-wjdr"
server_key: "bf265a83aea2688af8ad088283e21a29"
server_key_iv: "1551b264527c9805761425ccf135ec1a"
server_key_version: "encryptionkey@0"
I20260925 22:36:38.581489 15676 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 22:36:38.583876 15691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:38.585137 15676 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:36:38.585264 15676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-2/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-2/wal
uuid: "027d99d1e6ec4a54b31ddb654d0b78a6"
format_stamp: "Formatted at 2026-09-25 22:36:38 on dist-test-slave-wjdr"
server_key: "bf265a83aea2688af8ad088283e21a29"
server_key_iv: "1551b264527c9805761425ccf135ec1a"
server_key_version: "encryptionkey@0"
I20260925 22:36:38.585379 15676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:36:38.601416 15676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:36:38.604327 15676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:36:38.604570 15676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:36:38.605158 15676 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:36:38.605957 15676 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:36:38.606025 15676 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:38.606093 15676 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:36:38.606141 15676 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:38.614945 15676 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.195:44621
I20260925 22:36:38.614969 15804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.195:44621 every 8 connection(s)
I20260925 22:36:38.615832 15676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
I20260925 22:36:38.624351 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 15676
I20260925 22:36:38.624526 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-2/wal/instance
I20260925 22:36:38.624833 7131 external_mini_cluster.cc:1428] Setting key 950c70a9848842a0d28722a8a9c83003
Sep 25 22:36:38 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375798, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.195@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
I20260925 22:36:38.629832 15385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:38.617699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.195:55319 (local address 127.6.246.254:34777)
0925 22:36:38.617894 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:38.617898 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:38.618673 (+ 775us) server_negotiation.cc:407] Connection header received
0925 22:36:38.620059 (+ 1386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:38.620062 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:38.620122 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:38.620229 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:38.621430 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:38.621961 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:38.623122 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:38.623342 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:38.623455 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:38.626390 (+ 2935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:38.626397 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:38.626401 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:38.626431 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:38.628028 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:38.628545 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:38.628548 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:38.628549 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:38.628624 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:38.628935 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:38.628937 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:38.628938 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:38.629101 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:38.629172 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:38.629626 (+ 454us) server_negotiation.cc:299] Negotiation successful
0925 22:36:38.629717 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 22:36:38.630589 15807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:38.618044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:34777 (local address 127.6.246.195:55319)
0925 22:36:38.618519 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:38.618560 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:38.619842 (+ 1282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:38.620406 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:38.620413 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:38.620818 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:38.621273 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:38.621285 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:38.622084 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:38.622088 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:38.622980 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:38.622989 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:38.623491 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:38.623495 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:38.623660 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:38.624313 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:38.624330 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:38.626222 (+ 1892us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:36:38.628136 (+ 1914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:38.628140 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:38.628152 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:38.628404 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:38.628704 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:38.628706 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:38.628707 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:38.628844 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:38.629186 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:36:38.629191 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:36:38.629421 (+ 230us) client_negotiation.cc:769] Sending connection context
0925 22:36:38.629566 (+ 145us) client_negotiation.cc:240] Negotiation successful
0925 22:36:38.629800 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":143,"threads_started":1}
I20260925 22:36:38.631698 15805 heartbeater.cc:347] Connected to a master server at 127.6.246.254:34777
I20260925 22:36:38.632014 15805 heartbeater.cc:464] Registering TS with master...
I20260925 22:36:38.632551 15805 heartbeater.cc:511] Master 127.6.246.254:34777 requested a full tablet report, sending...
I20260925 22:36:38.633538 15346 ts_manager.cc:194] Registered new tserver with Master: 027d99d1e6ec4a54b31ddb654d0b78a6 (127.6.246.195:44621)
I20260925 22:36:38.634078 15346 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.246.195@KRBTEST.COM'} at 127.6.246.195:55319
I20260925 22:36:38.638758 7131 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.6.246.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.246.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.246.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/krb5kdc/oryx_127.6.246.193.keytab.
Entry for principal oryx/127.6.246.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/krb5kdc/oryx_127.6.246.193.keytab.
WARNING: no policy specified for oryx/127.6.246.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.246.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.246.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/krb5kdc/oryx_127.6.246.194.keytab.
Entry for principal oryx/127.6.246.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/krb5kdc/oryx_127.6.246.194.keytab.
WARNING: no policy specified for oryx/127.6.246.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.246.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.246.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/krb5kdc/oryx_127.6.246.195.keytab.
Entry for principal oryx/127.6.246.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/krb5kdc/oryx_127.6.246.195.keytab.
I20260925 22:36:38.711474 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 15408
I20260925 22:36:38.717417 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 15542
I20260925 22:36:38.722986 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 15676
I20260925 22:36:38.728152 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 15314
I20260925 22:36:38.734411 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:34777
--webserver_interface=127.6.246.254
--webserver_port=33301
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab
--principal=kudu/127.6.246.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:46577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.246.254:34777
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:36:38.843704 15814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:36:38.843981 15814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:36:38.844022 15814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:36:38.848042 15814 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:36:38.848098 15814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:36:38.848119 15814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:36:38.848135 15814 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:36:38.848204 15814 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:36:38.853410 15814 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:46577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.246.254:34777
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab
--principal=kudu/127.6.246.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:34777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.246.254
--webserver_port=33301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:36:38.854447 15814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:36:38.855443 15814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:36:38.863871 15819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:38.863989 15814 server_base.cc:1034] running on GCE node
W20260925 22:36:38.863871 15820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:38.863912 15822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:38.864684 15814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:36:38.865733 15814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:36:38.866925 15814 hybrid_clock.cc:648] HybridClock initialized: now 1790375798866933 us; error 53 us; skew 500 ppm
Sep 25 22:36:38 dist-test-slave-wjdr krb5kdc[15301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375798, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:36:38.869804 15814 init.cc:377] Logged in from keytab as kudu/127.6.246.254@KRBTEST.COM (short username kudu)
I20260925 22:36:38.870636 15814 webserver.cc:460] Webserver started at http://127.6.246.254:33301/ using document root <none> and password file <none>
I20260925 22:36:38.871129 15814 fs_manager.cc:359] Metadata directory not provided
I20260925 22:36:38.871199 15814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:36:38.874909 15814 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260925 22:36:38.876947 15829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:38.878113 15814 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:36:38.878208 15814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/master-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/master-0/wal
uuid: "7e697e1ddfb7436aa6aeb550f3015556"
format_stamp: "Formatted at 2026-09-25 22:36:37 on dist-test-slave-wjdr"
server_key: "457d6e014b750950044f99e06e7ffa77"
server_key_iv: "120dbf81f02ad4cdcec7dc5808a0308b"
server_key_version: "encryptionkey@0"
I20260925 22:36:38.878935 15814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:36:38.906654 15814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:36:38.909811 15814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:36:38.909956 15814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:36:38.916644 15814 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.254:34777
I20260925 22:36:38.916671 15881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.254:34777 every 8 connection(s)
I20260925 22:36:38.917675 15814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
I20260925 22:36:38.920564 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 15814
I20260925 22:36:38.921077 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.193:38715
--local_ip_for_outbound_sockets=127.6.246.193
--tserver_master_addrs=127.6.246.254:34777
--webserver_port=38581
--webserver_interface=127.6.246.193
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.193.keytab
--principal=kudu/127.6.246.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:46577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.6.246.193
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/krb5kdc/oryx_127.6.246.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:36:38.922019 15882 sys_catalog.cc:263] Verifying existing consensus state
I20260925 22:36:38.923610 15882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e697e1ddfb7436aa6aeb550f3015556: Bootstrap starting.
I20260925 22:36:38.939922 15882 log.cc:824] T 00000000000000000000000000000000 P 7e697e1ddfb7436aa6aeb550f3015556: Log is configured to *not* fsync() on all Append() calls
I20260925 22:36:38.946588 15882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e697e1ddfb7436aa6aeb550f3015556: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 22:36:38.946949 15882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e697e1ddfb7436aa6aeb550f3015556: Bootstrap complete.
I20260925 22:36:38.951048 15882 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e697e1ddfb7436aa6aeb550f3015556 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e697e1ddfb7436aa6aeb550f3015556" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 34777 } }
I20260925 22:36:38.951467 15882 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e697e1ddfb7436aa6aeb550f3015556 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e697e1ddfb7436aa6aeb550f3015556, State: Initialized, Role: FOLLOWER
I20260925 22:36:38.951918 15882 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e697e1ddfb7436aa6aeb550f3015556 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e697e1ddfb7436aa6aeb550f3015556" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 34777 } }
I20260925 22:36:38.952062 15882 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e697e1ddfb7436aa6aeb550f3015556 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:36:38.952140 15882 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e697e1ddfb7436aa6aeb550f3015556 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:36:38.952252 15882 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e697e1ddfb7436aa6aeb550f3015556 [term 1 FOLLOWER]: Advancing to term 2
I20260925 22:36:38.954146 15882 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e697e1ddfb7436aa6aeb550f3015556 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e697e1ddfb7436aa6aeb550f3015556" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 34777 } }
I20260925 22:36:38.954471 15882 leader_election.cc:302] T 00000000000000000000000000000000 P 7e697e1ddfb7436aa6aeb550f3015556 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e697e1ddfb7436aa6aeb550f3015556; no voters:
I20260925 22:36:38.954764 15882 leader_election.cc:288] T 00000000000000000000000000000000 P 7e697e1ddfb7436aa6aeb550f3015556 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 22:36:38.954977 15886 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e697e1ddfb7436aa6aeb550f3015556 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 22:36:38.955287 15886 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e697e1ddfb7436aa6aeb550f3015556 [term 2 LEADER]: Becoming Leader. State: Replica: 7e697e1ddfb7436aa6aeb550f3015556, State: Running, Role: LEADER
I20260925 22:36:38.955649 15886 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e697e1ddfb7436aa6aeb550f3015556 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e697e1ddfb7436aa6aeb550f3015556" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 34777 } }
I20260925 22:36:38.955943 15882 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e697e1ddfb7436aa6aeb550f3015556 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:36:38.957041 15887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e697e1ddfb7436aa6aeb550f3015556 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "7e697e1ddfb7436aa6aeb550f3015556" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e697e1ddfb7436aa6aeb550f3015556" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 34777 } } }
I20260925 22:36:38.957173 15887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e697e1ddfb7436aa6aeb550f3015556 [sys.catalog]: This master's current role is: LEADER
I20260925 22:36:38.957515 15888 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e697e1ddfb7436aa6aeb550f3015556 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e697e1ddfb7436aa6aeb550f3015556. Latest consensus state: current_term: 2 leader_uuid: "7e697e1ddfb7436aa6aeb550f3015556" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e697e1ddfb7436aa6aeb550f3015556" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 34777 } } }
I20260925 22:36:38.957674 15888 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e697e1ddfb7436aa6aeb550f3015556 [sys.catalog]: This master's current role is: LEADER
I20260925 22:36:38.961000 15902 catalog_manager.cc:1259] Loaded cluster ID: 633b4857a853448d9ecdd04d5310fde6
I20260925 22:36:38.961058 15902 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7e697e1ddfb7436aa6aeb550f3015556: loading cluster ID for follower catalog manager: success
I20260925 22:36:38.963369 15902 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7e697e1ddfb7436aa6aeb550f3015556: acquiring CA information for follower catalog manager: success
I20260925 22:36:38.964735 15902 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7e697e1ddfb7436aa6aeb550f3015556: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 22:36:38.964879 15895 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:36:38.965723 15895 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:36:38.966148 15895 catalog_manager.cc:1259] Loaded cluster ID: 633b4857a853448d9ecdd04d5310fde6
I20260925 22:36:38.966207 15895 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:36:38.966877 15895 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:36:38.967320 15895 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7e697e1ddfb7436aa6aeb550f3015556: Loaded TSK: 0
I20260925 22:36:38.968016 15895 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 22:36:39.034902 15884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:36:39.035180 15884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:36:39.035252 15884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:36:39.039449 15884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:36:39.039526 15884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:36:39.039613 15884 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.193
I20260925 22:36:39.044749 15884 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:46577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/krb5kdc/oryx_127.6.246.193.keytab
--principal=oryx/127.6.246.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.193:38715
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.246.193
--webserver_port=38581
--tserver_master_addrs=127.6.246.254:34777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15884
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.193
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:36:39.045673 15884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:36:39.046676 15884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:36:39.056967 15908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:39.057039 15911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:39.056975 15909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:39.057106 15884 server_base.cc:1034] running on GCE node
I20260925 22:36:39.057775 15884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:36:39.058379 15884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:36:39.059551 15884 hybrid_clock.cc:648] HybridClock initialized: now 1790375799059527 us; error 45 us; skew 500 ppm
Sep 25 22:36:39 dist-test-slave-wjdr krb5kdc[15301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375799, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.246.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:36:39.062570 15884 init.cc:377] Logged in from keytab as oryx/127.6.246.193@KRBTEST.COM (short username oryx)
I20260925 22:36:39.063589 15884 webserver.cc:460] Webserver started at http://127.6.246.193:38581/ using document root <none> and password file <none>
I20260925 22:36:39.064247 15884 fs_manager.cc:359] Metadata directory not provided
I20260925 22:36:39.064365 15884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:36:39.069172 15884 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:36:39.071573 15918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:39.072839 15884 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:36:39.072990 15884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-0/wal
uuid: "a7b2ec64e6a445b49615c39f03d5f0b6"
format_stamp: "Formatted at 2026-09-25 22:36:38 on dist-test-slave-wjdr"
server_key: "0d574358439dd712463f0e2d1845f506"
server_key_iv: "365edb238ff469dfd22737c3035b7f16"
server_key_version: "encryptionkey@0"
I20260925 22:36:39.073526 15884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:36:39.085884 15884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:36:39.088932 15884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:36:39.089123 15884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:36:39.089668 15884 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:36:39.090441 15884 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:36:39.090507 15884 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:39.090570 15884 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:36:39.090617 15884 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:39.100857 15884 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.193:38715
I20260925 22:36:39.100890 16031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.193:38715 every 8 connection(s)
I20260925 22:36:39.101809 15884 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
I20260925 22:36:39.107252 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 15884
I20260925 22:36:39.107676 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.194:38425
--local_ip_for_outbound_sockets=127.6.246.194
--tserver_master_addrs=127.6.246.254:34777
--webserver_port=35893
--webserver_interface=127.6.246.194
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.194.keytab
--principal=kudu/127.6.246.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:46577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.6.246.194
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/krb5kdc/oryx_127.6.246.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 22:36:39 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.193@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:36:39 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.193@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:36:39.115480 16035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:39.104140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:34777 (local address 127.6.246.193:43723)
0925 22:36:39.104856 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:39.104908 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:39.105878 (+ 970us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:39.106901 (+ 1023us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:39.106908 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:39.107263 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:39.107797 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:39.107820 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:39.109356 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:39.109359 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:39.110501 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:39.110516 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:39.111311 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:39.111326 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:39.111564 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:39.112166 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:39.112194 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:39.114814 (+ 2620us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.246.254:34777: Server oryx/127.6.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":150,"threads_started":1}
W20260925 22:36:39.115926 16032 heartbeater.cc:644] Failed to heartbeat to 127.6.246.254:34777 (0 consecutive failures): Not authorized: Failed to ping master at 127.6.246.254:34777: Client connection negotiation failed: client connection to 127.6.246.254:34777: Server oryx/127.6.246.254@KRBTEST.COM not found in Kerberos database
W20260925 22:36:39.116036 16034 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:36:39.104026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.193:43723 (local address 127.6.246.254:34777)
0925 22:36:39.104521 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:39.104541 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:39.105034 (+ 493us) server_negotiation.cc:407] Connection header received
0925 22:36:39.106268 (+ 1234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:39.106278 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:39.106555 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:39.106806 (+ 251us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:39.107984 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:39.109194 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:39.110636 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:39.111121 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:39.111295 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:39.115091 (+ 3796us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.246.193:43723: BlockingRecv error: recv got EOF from 127.6.246.193:43723 (error 108)
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":107,"threads_started":1}
Sep 25 22:36:39 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.193@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:36:39 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.193@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:36:39.122093 16035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:39.116560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:34777 (local address 127.6.246.193:51469)
0925 22:36:39.116778 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:39.116822 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:39.116891 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:39.117152 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:39.117154 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:39.117274 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:39.117368 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:39.117372 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:39.118473 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:39.118475 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:39.119016 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:39.119024 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:39.119601 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:39.119604 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:39.119670 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:39.120138 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:39.120149 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:39.122004 (+ 1855us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.246.254:34777: Server oryx/127.6.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163}
W20260925 22:36:39.122257 16034 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:36:39.116671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.193:51469 (local address 127.6.246.254:34777)
0925 22:36:39.116808 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:39.116815 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:39.116866 (+ 51us) server_negotiation.cc:407] Connection header received
0925 22:36:39.116979 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:39.116984 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:39.117043 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:39.117142 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:39.117476 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:39.118336 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:39.119129 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:39.119485 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:39.119581 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:39.122153 (+ 2572us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.246.193:51469: BlockingRecv error: recv got EOF from 127.6.246.193:51469 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 25 22:36:39 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.193@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:36:39 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.193@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:36:39.127563 16035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:39.122735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:34777 (local address 127.6.246.193:47619)
0925 22:36:39.122907 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:39.122923 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:39.122979 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:39.123216 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:39.123218 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:39.123331 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:39.123394 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:39.123398 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:39.124334 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:39.124336 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:39.124877 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:39.124885 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:39.125456 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:39.125458 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:39.125512 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:39.125939 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:39.125950 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:39.127480 (+ 1530us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.246.254:34777: Server oryx/127.6.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103}
W20260925 22:36:39.127722 16034 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:36:39.122828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.193:47619 (local address 127.6.246.254:34777)
0925 22:36:39.122932 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:39.122935 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:39.122950 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:36:39.123072 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:39.123075 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:39.123116 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:39.123200 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:39.123483 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:39.124241 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:39.124986 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:39.125357 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:39.125461 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:39.127622 (+ 2161us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.246.193:47619: BlockingRecv error: recv got EOF from 127.6.246.193:47619 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260925 22:36:39.127826 16032 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 22:36:39.218580 16036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:36:39.218856 16036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:36:39.218938 16036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:36:39.222966 16036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:36:39.223038 16036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:36:39.223147 16036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.194
I20260925 22:36:39.228086 16036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:46577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/krb5kdc/oryx_127.6.246.194.keytab
--principal=oryx/127.6.246.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.194:38425
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.246.194
--webserver_port=35893
--tserver_master_addrs=127.6.246.254:34777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16036
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.194
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:36:39.229059 16036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:36:39.230000 16036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:36:39.238833 16042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:39.238971 16041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:39.239171 16036 server_base.cc:1034] running on GCE node
W20260925 22:36:39.239300 16044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:39.239661 16036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:36:39.240232 16036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:36:39.241416 16036 hybrid_clock.cc:648] HybridClock initialized: now 1790375799241396 us; error 33 us; skew 500 ppm
Sep 25 22:36:39 dist-test-slave-wjdr krb5kdc[15301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375799, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.246.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:36:39.244051 16036 init.cc:377] Logged in from keytab as oryx/127.6.246.194@KRBTEST.COM (short username oryx)
I20260925 22:36:39.244848 16036 webserver.cc:460] Webserver started at http://127.6.246.194:35893/ using document root <none> and password file <none>
I20260925 22:36:39.245325 16036 fs_manager.cc:359] Metadata directory not provided
I20260925 22:36:39.245398 16036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:36:39.248867 16036 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:36:39.250759 16051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:39.251945 16036 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:36:39.252055 16036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-1/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-1/wal
uuid: "006ee5a095c443949c0baa2ffd9682d3"
format_stamp: "Formatted at 2026-09-25 22:36:38 on dist-test-slave-wjdr"
server_key: "69ee9e8688707525b1ee332373289d84"
server_key_iv: "52a97f6077625000aaac38b31c2d88e9"
server_key_version: "encryptionkey@0"
I20260925 22:36:39.252609 16036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:36:39.277537 16036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:36:39.280625 16036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:36:39.280786 16036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:36:39.281337 16036 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:36:39.282238 16036 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:36:39.282299 16036 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:39.282368 16036 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:36:39.282421 16036 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:39.291002 16036 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.194:38425
I20260925 22:36:39.291029 16164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.194:38425 every 8 connection(s)
I20260925 22:36:39.291947 16036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
I20260925 22:36:39.293798 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 16036
I20260925 22:36:39.294286 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.195:44621
--local_ip_for_outbound_sockets=127.6.246.195
--tserver_master_addrs=127.6.246.254:34777
--webserver_port=41243
--webserver_interface=127.6.246.195
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.195.keytab
--principal=kudu/127.6.246.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:46577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.6.246.195
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/krb5kdc/oryx_127.6.246.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 22:36:39 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.194@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:36:39 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.194@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
W20260925 22:36:39.303164 16034 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:36:39.293655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.194:36421 (local address 127.6.246.254:34777)
0925 22:36:39.293809 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:39.293812 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:39.295434 (+ 1622us) server_negotiation.cc:407] Connection header received
0925 22:36:39.296306 (+ 872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:39.296309 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:39.296374 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:39.296510 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:39.297607 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:39.298130 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:39.299234 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:39.299460 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:39.299568 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:39.303040 (+ 3472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.246.194:36421: BlockingRecv error: recv got EOF from 127.6.246.194:36421 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 22:36:39.303735 16167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:39.294212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:34777 (local address 127.6.246.194:36421)
0925 22:36:39.295283 (+ 1071us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:39.295323 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:39.296111 (+ 788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:39.296681 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:39.296687 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:39.297024 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:39.297468 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:39.297481 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:39.298249 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:39.298253 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:39.299109 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:39.299118 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:39.299604 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:39.299608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:39.299768 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:39.300378 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:39.300395 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:39.302933 (+ 2538us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.246.254:34777: Server oryx/127.6.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":888,"thread_start_us":728,"threads_started":1}
W20260925 22:36:39.304168 16165 heartbeater.cc:644] Failed to heartbeat to 127.6.246.254:34777 (0 consecutive failures): Not authorized: Failed to ping master at 127.6.246.254:34777: Client connection negotiation failed: client connection to 127.6.246.254:34777: Server oryx/127.6.246.254@KRBTEST.COM not found in Kerberos database
Sep 25 22:36:39 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.194@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:36:39 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.194@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:36:39.309404 16167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:39.304639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:34777 (local address 127.6.246.194:48425)
0925 22:36:39.304744 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:39.304759 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:39.304834 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:39.305071 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:39.305074 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:39.305225 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:39.305308 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:39.305313 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:39.306005 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:39.306011 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:39.306541 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:39.306548 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:39.306969 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:39.306972 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:39.307059 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:39.307524 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:39.307534 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:39.309271 (+ 1737us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.246.254:34777: Server oryx/127.6.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":30}
W20260925 22:36:39.309501 16034 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:36:39.304693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.194:48425 (local address 127.6.246.254:34777)
0925 22:36:39.304805 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:39.304810 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:39.304823 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:36:39.304933 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:39.304936 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:39.304978 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:39.305058 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:39.305424 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:39.305907 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:39.306652 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:39.306866 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:39.306970 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:39.309388 (+ 2418us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.246.194:48425: BlockingRecv error: recv got EOF from 127.6.246.194:48425 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 25 22:36:39 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.194@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:36:39 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.194@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:36:39.314553 16167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:39.309987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:34777 (local address 127.6.246.194:46409)
0925 22:36:39.310082 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:39.310097 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:39.310167 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:39.310377 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:39.310386 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:39.310530 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:39.310593 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:39.310597 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:39.311289 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:39.311294 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:39.311796 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:39.311803 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:39.312228 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:39.312231 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:39.312304 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:39.312745 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:39.312755 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:39.314425 (+ 1670us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.246.254:34777: Server oryx/127.6.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 22:36:39.314632 16034 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:36:39.310074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.194:46409 (local address 127.6.246.254:34777)
0925 22:36:39.310182 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:39.310189 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:39.310204 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:36:39.310240 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:39.310243 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:39.310285 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:39.310363 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:39.310698 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:39.311183 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:39.311919 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:39.312133 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:39.312227 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:39.314526 (+ 2299us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.246.194:46409: BlockingRecv error: recv got EOF from 127.6.246.194:46409 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260925 22:36:39.314774 16165 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 22:36:39.406101 16168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:36:39.406340 16168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:36:39.406386 16168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:36:39.410501 16168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:36:39.410552 16168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:36:39.410645 16168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.195
I20260925 22:36:39.415725 16168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:46577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/krb5kdc/oryx_127.6.246.195.keytab
--principal=oryx/127.6.246.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.195:44621
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.246.195
--webserver_port=41243
--tserver_master_addrs=127.6.246.254:34777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16168
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.195
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:36:39.416724 16168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:36:39.417672 16168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:36:39.426892 16176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:39.426949 16168 server_base.cc:1034] running on GCE node
W20260925 22:36:39.426899 16173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:39.426931 16174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:39.427533 16168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:36:39.428131 16168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:36:39.429303 16168 hybrid_clock.cc:648] HybridClock initialized: now 1790375799429289 us; error 26 us; skew 500 ppm
Sep 25 22:36:39 dist-test-slave-wjdr krb5kdc[15301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375799, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.246.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:36:39.431841 16168 init.cc:377] Logged in from keytab as oryx/127.6.246.195@KRBTEST.COM (short username oryx)
I20260925 22:36:39.432744 16168 webserver.cc:460] Webserver started at http://127.6.246.195:41243/ using document root <none> and password file <none>
I20260925 22:36:39.433269 16168 fs_manager.cc:359] Metadata directory not provided
I20260925 22:36:39.433365 16168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:36:39.436901 16168 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:36:39.438819 16183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:39.439939 16168 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:36:39.440066 16168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-2/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-2/wal
uuid: "027d99d1e6ec4a54b31ddb654d0b78a6"
format_stamp: "Formatted at 2026-09-25 22:36:38 on dist-test-slave-wjdr"
server_key: "bf265a83aea2688af8ad088283e21a29"
server_key_iv: "1551b264527c9805761425ccf135ec1a"
server_key_version: "encryptionkey@0"
I20260925 22:36:39.440598 16168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:36:39.460297 16168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:36:39.462879 16168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:36:39.463055 16168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:36:39.463577 16168 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:36:39.464358 16168 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:36:39.464422 16168 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:39.464507 16168 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:36:39.464555 16168 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:39.473294 16168 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.195:44621
I20260925 22:36:39.473317 16296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.195:44621 every 8 connection(s)
I20260925 22:36:39.474255 16168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
I20260925 22:36:39.480636 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 16168
Sep 25 22:36:39 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.195@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:36:39 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.195@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
W20260925 22:36:39.484814 16034 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:36:39.475843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.195:35065 (local address 127.6.246.254:34777)
0925 22:36:39.476001 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:39.476005 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:39.477050 (+ 1045us) server_negotiation.cc:407] Connection header received
0925 22:36:39.477887 (+ 837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:39.477890 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:39.477934 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:39.478005 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:39.479041 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:39.479547 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:39.480686 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:39.480959 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:39.481068 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:39.484717 (+ 3649us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.246.195:35065: BlockingRecv error: recv got EOF from 127.6.246.195:35065 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260925 22:36:39.485510 16299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:39.476227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:34777 (local address 127.6.246.195:35065)
0925 22:36:39.476924 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:39.476961 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:39.477663 (+ 702us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:39.478150 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:39.478157 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:39.478492 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:39.478924 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:39.478937 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:39.479643 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:39.479646 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:39.480570 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:39.480578 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:39.481101 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:39.481105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:39.481276 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:39.481920 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:39.481947 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:39.484630 (+ 2683us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.246.254:34777: Server oryx/127.6.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":109,"threads_started":1}
W20260925 22:36:39.485929 16297 heartbeater.cc:644] Failed to heartbeat to 127.6.246.254:34777 (0 consecutive failures): Not authorized: Failed to ping master at 127.6.246.254:34777: Client connection negotiation failed: client connection to 127.6.246.254:34777: Server oryx/127.6.246.254@KRBTEST.COM not found in Kerberos database
Sep 25 22:36:39 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.195@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:36:39 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.195@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:36:39.491446 16299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:39.486322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:34777 (local address 127.6.246.195:60541)
0925 22:36:39.486419 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:39.486432 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:39.486503 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:39.486716 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:39.486718 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:39.486864 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:39.486946 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:39.486951 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:39.487720 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:39.487723 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:39.488526 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:39.488535 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:39.489058 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:39.489060 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:39.489124 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:39.489528 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:39.489541 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:39.491352 (+ 1811us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.246.254:34777: Server oryx/127.6.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 22:36:39.491541 16034 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:36:39.486407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.195:60541 (local address 127.6.246.254:34777)
0925 22:36:39.486505 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:39.486509 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:39.486523 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:36:39.486581 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:39.486585 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:39.486625 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:39.486702 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:39.487049 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:39.487617 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:39.488631 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:39.488958 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:39.489062 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:39.491455 (+ 2393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.246.195:60541: BlockingRecv error: recv got EOF from 127.6.246.195:60541 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 22:36:39.491546 16301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:39.481616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42878 (local address 127.6.246.254:34777)
0925 22:36:39.481818 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:39.481821 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:39.482013 (+ 192us) server_negotiation.cc:407] Connection header received
0925 22:36:39.482122 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:39.482124 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:39.482160 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:39.482229 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:39.483602 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:39.484220 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:39.485374 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:39.485626 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:39.485740 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:39.487451 (+ 1711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:39.487457 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:39.487473 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:39.487502 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:39.489391 (+ 1889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:39.489800 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:39.489806 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:39.489813 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:39.489976 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:39.490276 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:39.490278 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:39.490279 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:39.490630 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:39.490818 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:39.491128 (+ 310us) server_negotiation.cc:299] Negotiation successful
0925 22:36:39.491369 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":76,"threads_started":1}
Sep 25 22:36:39 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.195@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:36:39 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.195@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:36:39.496909 16299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:39.492322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:34777 (local address 127.6.246.195:44433)
0925 22:36:39.492417 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:39.492428 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:39.492538 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:39.492838 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:39.492840 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:39.492985 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:39.493061 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:39.493066 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:39.493923 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:39.493927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:39.494565 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:39.494573 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:39.494962 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:39.494964 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:39.495016 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:39.495335 (+ 319us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:39.495345 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:39.496832 (+ 1487us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.246.254:34777: Server oryx/127.6.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":39}
W20260925 22:36:39.497095 16034 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:36:39.492554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.195:44433 (local address 127.6.246.254:34777)
0925 22:36:39.492671 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:39.492674 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:39.492685 (+ 11us) server_negotiation.cc:407] Connection header received
0925 22:36:39.492712 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:39.492714 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:39.492742 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:39.492803 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:39.493168 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:39.493793 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:39.494658 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:39.494877 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:39.494952 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:39.496970 (+ 2018us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.246.195:44433: BlockingRecv error: recv got EOF from 127.6.246.195:44433 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260925 22:36:39.497205 16297 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 25 22:36:40 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.193@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:36:40 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.193@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:36:40.134030 16302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:40.128447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:34777 (local address 127.6.246.193:53901)
0925 22:36:40.128773 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:40.128789 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:40.128937 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:40.129225 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:40.129228 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:40.129429 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:40.129548 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:40.129555 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:40.130318 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:40.130323 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:40.130914 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:40.130924 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:40.131402 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:40.131405 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:40.131504 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:40.132042 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:40.132057 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:40.133914 (+ 1857us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.246.254:34777: Server oryx/127.6.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":71,"threads_started":1}
W20260925 22:36:40.134303 16303 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:36:40.128526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.193:53901 (local address 127.6.246.254:34777)
0925 22:36:40.128785 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:40.128790 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:40.128901 (+ 111us) server_negotiation.cc:407] Connection header received
0925 22:36:40.129054 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:40.129057 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:40.129111 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:40.129202 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:40.129669 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:40.130205 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:40.131039 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:40.131283 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:40.131395 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:40.134048 (+ 2653us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.246.193:53901: BlockingRecv error: recv got EOF from 127.6.246.193:53901 (error 108)
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":70,"threads_started":1}
Sep 25 22:36:40 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.194@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:36:40 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.194@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:36:40.320297 16304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:40.315355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:34777 (local address 127.6.246.194:54337)
0925 22:36:40.315625 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:40.315639 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:40.315731 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:40.315959 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:40.315961 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:40.316139 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:40.316255 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:40.316260 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:40.316984 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:40.316986 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:40.317550 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:40.317557 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:40.317967 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:40.317969 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:40.318042 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:40.318505 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:40.318516 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:40.320185 (+ 1669us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.246.254:34777: Server oryx/127.6.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":83,"threads_started":1}
W20260925 22:36:40.320508 16303 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:36:40.315419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.194:54337 (local address 127.6.246.254:34777)
0925 22:36:40.315581 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:40.315585 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:40.315703 (+ 118us) server_negotiation.cc:407] Connection header received
0925 22:36:40.315805 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:40.315807 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:40.315849 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:40.315923 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:40.316344 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:40.316896 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:40.317653 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:40.317869 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:40.317948 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:40.320344 (+ 2396us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.246.194:54337: BlockingRecv error: recv got EOF from 127.6.246.194:54337 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 25 22:36:40 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.195@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:36:40 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.195@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:36:40.502774 16305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:40.497779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:34777 (local address 127.6.246.195:39137)
0925 22:36:40.498016 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:40.498029 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:40.498123 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:40.498342 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:40.498344 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:40.498514 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:40.498604 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:40.498609 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:40.499281 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:40.499284 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:40.499821 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:40.499828 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:40.500217 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:40.500219 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:40.500279 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:40.500793 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:40.500805 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:40.502618 (+ 1813us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.246.254:34777: Server oryx/127.6.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":78,"threads_started":1}
W20260925 22:36:40.502913 16303 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:36:40.497824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.195:39137 (local address 127.6.246.254:34777)
0925 22:36:40.497960 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:40.497964 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:40.498094 (+ 130us) server_negotiation.cc:407] Connection header received
0925 22:36:40.498198 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:40.498200 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:40.498244 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:40.498310 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:40.498693 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:40.499197 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:40.499920 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:40.500128 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:40.500202 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:40.502758 (+ 2556us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.246.195:39137: BlockingRecv error: recv got EOF from 127.6.246.195:39137 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 25 22:36:41 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.193@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:36:41 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.193@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:36:41.141398 16306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:41.134970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:34777 (local address 127.6.246.193:52043)
0925 22:36:41.135336 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:41.135355 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:41.135506 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:41.135826 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:41.135828 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:41.136045 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:41.136158 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:41.136164 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:41.137045 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:41.137049 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:41.137677 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:41.137687 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:41.138175 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:41.138178 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:41.138254 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:41.138940 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:41.138969 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:41.141233 (+ 2264us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.246.254:34777: Server oryx/127.6.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":111,"threads_started":1}
W20260925 22:36:41.141547 16307 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:36:41.134996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.193:52043 (local address 127.6.246.254:34777)
0925 22:36:41.135379 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:41.135384 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:41.135422 (+ 38us) server_negotiation.cc:407] Connection header received
0925 22:36:41.135632 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:41.135636 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:41.135703 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:41.135785 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:41.136264 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:41.136917 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:41.137793 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:41.138038 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:41.138149 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:41.141390 (+ 3241us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.246.193:52043: BlockingRecv error: recv got EOF from 127.6.246.193:52043 (error 108)
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":154,"threads_started":1}
Sep 25 22:36:41 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.194@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:36:41 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.194@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:36:41.327747 16308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:41.321396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:34777 (local address 127.6.246.194:45897)
0925 22:36:41.321763 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:41.321778 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:41.321919 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:41.322250 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:41.322256 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:41.322494 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:41.322659 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:41.322668 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:41.323721 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:41.323724 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:41.324318 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:41.324326 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:41.324786 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:41.324789 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:41.324863 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:41.325452 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:41.325464 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:41.327588 (+ 2124us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.246.254:34777: Server oryx/127.6.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":159,"threads_started":1}
W20260925 22:36:41.327900 16307 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:36:41.321493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.194:45897 (local address 127.6.246.254:34777)
0925 22:36:41.321695 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:41.321700 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:41.321885 (+ 185us) server_negotiation.cc:407] Connection header received
0925 22:36:41.322023 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:41.322026 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:41.322095 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:41.322196 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:41.322789 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:41.323610 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:41.324426 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:41.324680 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:41.324765 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:41.327739 (+ 2974us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.246.194:45897: BlockingRecv error: recv got EOF from 127.6.246.194:45897 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 25 22:36:41 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.195@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:36:41 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.195@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:36:41.510385 16309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:41.503797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:34777 (local address 127.6.246.195:58847)
0925 22:36:41.504164 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:41.504183 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:41.504323 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:41.504675 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:41.504678 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:41.504967 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:41.505119 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:41.505126 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:41.505953 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:41.505955 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:41.506546 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:41.506553 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:41.507096 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:41.507099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:41.507169 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:41.507747 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:41.507758 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:41.510159 (+ 2401us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.246.254:34777: Server oryx/127.6.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":134,"threads_started":1}
W20260925 22:36:41.510419 16307 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:36:41.503967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.195:58847 (local address 127.6.246.254:34777)
0925 22:36:41.504217 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:41.504224 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:41.504247 (+ 23us) server_negotiation.cc:407] Connection header received
0925 22:36:41.504434 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:41.504438 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:41.504542 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:41.504634 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:41.505268 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:41.505852 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:41.506670 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:41.506942 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:41.507086 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:41.510303 (+ 3217us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.246.195:58847: BlockingRecv error: recv got EOF from 127.6.246.195:58847 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 25 22:36:42 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.193@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:36:42 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.193@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:36:42.148974 16310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:42.142472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:34777 (local address 127.6.246.193:36143)
0925 22:36:42.142794 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:42.142810 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:42.142926 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:42.143292 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:42.143294 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:42.143579 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:42.143709 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:42.143716 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:42.144593 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:42.144596 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:42.145416 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:42.145426 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:42.145985 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:42.145989 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:42.146077 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:42.146611 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:42.146625 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:42.148821 (+ 2196us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.246.254:34777: Server oryx/127.6.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":110,"threads_started":1}
W20260925 22:36:42.149226 16311 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:36:42.142537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.193:36143 (local address 127.6.246.254:34777)
0925 22:36:42.142869 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:42.142873 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:42.142889 (+ 16us) server_negotiation.cc:407] Connection header received
0925 22:36:42.143068 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:42.143071 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:42.143127 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:42.143217 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:42.143836 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:42.144429 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:42.145535 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:42.145852 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:42.145968 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:42.148980 (+ 3012us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.246.193:36143: BlockingRecv error: recv got EOF from 127.6.246.193:36143 (error 108)
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":149,"threads_started":1}
Sep 25 22:36:42 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.194@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:36:42 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.194@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:36:42.335587 16312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:42.328788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:34777 (local address 127.6.246.194:57149)
0925 22:36:42.329179 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:42.329198 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:42.329367 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:42.329697 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:42.329702 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:42.329922 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:42.330048 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:42.330054 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:42.330939 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:42.330942 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:42.331641 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:42.331652 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:42.332430 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:42.332433 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:42.332562 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:42.333136 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:42.333150 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:42.335411 (+ 2261us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.246.254:34777: Server oryx/127.6.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":125,"threads_started":1}
W20260925 22:36:42.335774 16311 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:36:42.328868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.194:57149 (local address 127.6.246.254:34777)
0925 22:36:42.329053 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:42.329059 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:42.329294 (+ 235us) server_negotiation.cc:407] Connection header received
0925 22:36:42.329471 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:42.329474 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:42.329527 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:42.329637 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:42.330191 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:42.330823 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:42.331762 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:42.332293 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:42.332426 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:42.335600 (+ 3174us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.246.194:57149: BlockingRecv error: recv got EOF from 127.6.246.194:57149 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 25 22:36:42 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.195@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:36:42 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.195@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:36:42.520756 16313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:42.511393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:34777 (local address 127.6.246.195:50771)
0925 22:36:42.511706 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:42.511721 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:42.511839 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:42.512091 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:42.512093 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:42.512307 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:42.512421 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:42.512426 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:42.513251 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:42.513255 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:42.513884 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:42.513893 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:42.514312 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:42.514314 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:42.514379 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:42.518390 (+ 4011us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:42.518402 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:42.520584 (+ 2182us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.246.254:34777: Server oryx/127.6.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":115,"threads_started":1}
W20260925 22:36:42.520881 16311 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:36:42.511459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.195:50771 (local address 127.6.246.254:34777)
0925 22:36:42.511629 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:42.511633 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:42.511795 (+ 162us) server_negotiation.cc:407] Connection header received
0925 22:36:42.511925 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:42.511927 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:42.511981 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:42.512055 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:42.512564 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:42.513143 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:42.513990 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:42.514218 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:42.514300 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:42.520728 (+ 6428us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.246.195:50771: BlockingRecv error: recv got EOF from 127.6.246.195:50771 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 25 22:36:43 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.193@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:36:43 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.193@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:36:43.156025 16314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:43.149920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:34777 (local address 127.6.246.193:50501)
0925 22:36:43.150279 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:43.150297 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:43.150445 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:43.150724 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:43.150726 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:43.150973 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:43.151135 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:43.151144 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:43.151967 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:43.151970 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:43.152618 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:43.152628 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:43.153049 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:43.153071 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:43.153139 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:43.153705 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:43.153717 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:43.155845 (+ 2128us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.246.254:34777: Server oryx/127.6.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":92,"threads_started":1}
W20260925 22:36:43.156289 16315 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:36:43.149973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.193:50501 (local address 127.6.246.254:34777)
0925 22:36:43.150316 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:43.150319 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:43.150375 (+ 56us) server_negotiation.cc:407] Connection header received
0925 22:36:43.150532 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:43.150534 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:43.150586 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:43.150679 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:43.151245 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:43.151840 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:43.152717 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:43.152952 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:43.153031 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:43.155987 (+ 2956us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.246.193:50501: BlockingRecv error: recv got EOF from 127.6.246.193:50501 (error 108)
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":153,"threads_started":1}
Sep 25 22:36:43 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.194@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:36:43 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.194@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:36:43.343097 16316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:43.336634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:34777 (local address 127.6.246.194:36215)
0925 22:36:43.336988 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:43.337003 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:43.337157 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:43.337462 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:43.337465 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:43.337722 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:43.337853 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:43.337859 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:43.338664 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:43.338669 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:43.339282 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:43.339289 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:43.339758 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:43.339761 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:43.339858 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:43.340559 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:43.340571 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:43.342954 (+ 2383us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.246.254:34777: Server oryx/127.6.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":129,"threads_started":1}
W20260925 22:36:43.343300 16315 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:36:43.336698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.194:36215 (local address 127.6.246.254:34777)
0925 22:36:43.336883 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:43.336889 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:43.337086 (+ 197us) server_negotiation.cc:407] Connection header received
0925 22:36:43.337284 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:43.337286 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:43.337338 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:43.337430 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:43.337973 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:43.338557 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:43.339399 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:43.339637 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:43.339745 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:43.343150 (+ 3405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.246.194:36215: BlockingRecv error: recv got EOF from 127.6.246.194:36215 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 25 22:36:43 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.195@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:36:43 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.195@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:36:43.528677 16317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:43.521691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:34777 (local address 127.6.246.195:33371)
0925 22:36:43.522089 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:43.522109 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:43.522302 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:43.522655 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:43.522659 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:43.522919 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:43.523082 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:43.523090 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:43.523982 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:43.523988 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:43.524754 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:43.524772 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:43.525291 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:43.525295 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:43.525393 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:43.526001 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:43.526013 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:43.528409 (+ 2396us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.246.254:34777: Server oryx/127.6.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":97,"threads_started":1}
W20260925 22:36:43.528825 16315 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:36:43.521770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.195:33371 (local address 127.6.246.254:34777)
0925 22:36:43.521991 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:43.521997 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:43.522213 (+ 216us) server_negotiation.cc:407] Connection header received
0925 22:36:43.522428 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:43.522433 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:43.522502 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:43.522621 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:43.523220 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:43.523855 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:43.524893 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:43.525166 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:43.525290 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:43.528675 (+ 3385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.246.195:33371: BlockingRecv error: recv got EOF from 127.6.246.195:33371 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 25 22:36:44 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.193@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:36:44 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.193@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:36:44.163125 16318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:44.157099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:34777 (local address 127.6.246.193:37211)
0925 22:36:44.157401 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:44.157428 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:44.157522 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:44.157784 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:44.157787 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:44.158033 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:44.158296 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:44.158305 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:44.159188 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:44.159191 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:44.159790 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:44.159798 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:44.160232 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:44.160234 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:44.160299 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:44.160866 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:44.160879 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:44.162947 (+ 2068us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.246.254:34777: Server oryx/127.6.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":110,"threads_started":1}
W20260925 22:36:44.163192 16319 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:36:44.157172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.193:37211 (local address 127.6.246.254:34777)
0925 22:36:44.157473 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:44.157476 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:44.157489 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:36:44.157613 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:44.157615 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:44.157659 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:44.157734 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:44.158413 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:44.159079 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:44.159909 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:44.160130 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:44.160210 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:44.163065 (+ 2855us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.246.193:37211: BlockingRecv error: recv got EOF from 127.6.246.193:37211 (error 108)
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":65,"threads_started":1}
Sep 25 22:36:44 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.194@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:36:44 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.194@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:36:44.349808 16320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:44.344200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:34777 (local address 127.6.246.194:41109)
0925 22:36:44.344564 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:44.344582 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:44.344708 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:44.345005 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:44.345008 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:44.345222 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:44.345328 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:44.345334 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:44.346094 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:44.346098 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:44.346695 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:44.346704 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:44.347151 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:44.347155 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:44.347239 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:44.347767 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:44.347779 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:44.349635 (+ 1856us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.246.254:34777: Server oryx/127.6.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":104,"threads_started":1}
W20260925 22:36:44.349843 16319 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:36:44.344357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.194:41109 (local address 127.6.246.254:34777)
0925 22:36:44.344526 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:44.344530 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:44.344646 (+ 116us) server_negotiation.cc:407] Connection header received
0925 22:36:44.344802 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:44.344806 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:44.344885 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:44.344975 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:44.345457 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:44.345989 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:44.346809 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:44.347042 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:44.347147 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:44.349749 (+ 2602us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.246.194:41109: BlockingRecv error: recv got EOF from 127.6.246.194:41109 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 25 22:36:44 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.195@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:36:44 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.195@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:36:44.534973 16321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:44.529620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:34777 (local address 127.6.246.195:54569)
0925 22:36:44.529898 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:44.529935 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:44.530047 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:44.530267 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:44.530269 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:44.530470 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:44.530575 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:44.530580 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:44.531297 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:44.531299 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:44.531856 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:44.531862 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:44.532258 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:44.532260 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:44.532322 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:44.532952 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:44.532963 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:44.534851 (+ 1888us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.246.254:34777: Server oryx/127.6.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":108,"threads_started":1}
W20260925 22:36:44.535089 16319 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:36:44.529715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.195:54569 (local address 127.6.246.254:34777)
0925 22:36:44.529863 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:44.529867 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:44.530012 (+ 145us) server_negotiation.cc:407] Connection header received
0925 22:36:44.530125 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:44.530127 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:44.530171 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:44.530238 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:44.530679 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:44.531215 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:44.531951 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:44.532164 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:44.532240 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:44.534980 (+ 2740us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.246.195:54569: BlockingRecv error: recv got EOF from 127.6.246.195:54569 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 25 22:36:45 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.193@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:36:45 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.193@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:36:45.170742 16322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:45.164210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:34777 (local address 127.6.246.193:37385)
0925 22:36:45.164580 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:45.164598 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:45.164736 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:45.165115 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:45.165118 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:45.165359 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:45.165494 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:45.165500 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:45.166358 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:45.166362 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:45.167278 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:45.167291 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:45.167887 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:45.167890 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:45.167957 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:45.168565 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:45.168578 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:45.170603 (+ 2025us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.246.254:34777: Server oryx/127.6.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":120,"threads_started":1}
W20260925 22:36:45.170918 16323 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:36:45.164271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.193:37385 (local address 127.6.246.254:34777)
0925 22:36:45.164608 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:45.164612 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:45.164724 (+ 112us) server_negotiation.cc:407] Connection header received
0925 22:36:45.164846 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:45.164849 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:45.164927 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:45.165028 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:45.165662 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:45.166220 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:45.167431 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:45.167770 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:45.167855 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:45.170764 (+ 2909us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.246.193:37385: BlockingRecv error: recv got EOF from 127.6.246.193:37385 (error 108)
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":135,"threads_started":1}
Sep 25 22:36:45 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.194@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:36:45 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.194@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
W20260925 22:36:45.357225 16323 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:36:45.350778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.194:55961 (local address 127.6.246.254:34777)
0925 22:36:45.350942 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:45.350946 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:45.351033 (+ 87us) server_negotiation.cc:407] Connection header received
0925 22:36:45.351152 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:45.351154 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:45.351211 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:45.351318 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:45.351779 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:45.352629 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:45.353655 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:45.353909 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:45.353995 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:45.357118 (+ 3123us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.246.194:55961: BlockingRecv error: recv got EOF from 127.6.246.194:55961 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 22:36:45.357221 16324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:45.350643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:34777 (local address 127.6.246.194:55961)
0925 22:36:45.350940 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:45.350955 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:45.351062 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:45.351356 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:45.351359 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:45.351563 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:45.351668 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:45.351673 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:45.352773 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:45.352777 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:45.353548 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:45.353558 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:45.354014 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:45.354017 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:45.354082 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:45.354636 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:45.354648 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:45.356917 (+ 2269us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.246.254:34777: Server oryx/127.6.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":98,"threads_started":1}
Sep 25 22:36:45 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.195@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:36:45 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.195@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:36:45.542816 16325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:45.535977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:34777 (local address 127.6.246.195:37375)
0925 22:36:45.536364 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:45.536380 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:45.536547 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:45.536935 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:45.536938 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:45.537175 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:45.537288 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:45.537294 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:45.538127 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:45.538130 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:45.538771 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:45.538780 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:45.539407 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:45.539411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:45.539512 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:45.540196 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:45.540210 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:45.542595 (+ 2385us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.246.254:34777: Server oryx/127.6.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":126,"threads_started":1}
W20260925 22:36:45.543059 16323 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:36:45.536119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.195:37375 (local address 127.6.246.254:34777)
0925 22:36:45.536323 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:45.536328 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:45.536477 (+ 149us) server_negotiation.cc:407] Connection header received
0925 22:36:45.536647 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:45.536650 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:45.536712 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:45.536781 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:45.537418 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:45.538006 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:45.538902 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:45.539254 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:45.539387 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:45.542875 (+ 3488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.246.195:37375: BlockingRecv error: recv got EOF from 127.6.246.195:37375 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 25 22:36:46 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.193@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:36:46 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.193@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:36:46.178050 16326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:46.171696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:34777 (local address 127.6.246.193:34375)
0925 22:36:46.172060 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:46.172106 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:46.172216 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:46.172557 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:46.172560 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:46.172786 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:46.172908 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:46.172914 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:46.173821 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:46.173824 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:46.174641 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:46.174653 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:46.175126 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:46.175129 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:46.175205 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:46.175780 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:46.175793 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:46.177904 (+ 2111us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.246.254:34777: Server oryx/127.6.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":104,"threads_started":1}
W20260925 22:36:46.178202 16327 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:36:46.171737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.193:34375 (local address 127.6.246.254:34777)
0925 22:36:46.172141 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:46.172145 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:46.172158 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:36:46.172341 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:46.172344 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:46.172402 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:46.172532 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:46.173020 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:46.173686 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:46.174775 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:46.175009 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:46.175100 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:46.178051 (+ 2951us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.246.193:34375: BlockingRecv error: recv got EOF from 127.6.246.193:34375 (error 108)
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":243,"threads_started":1}
Sep 25 22:36:46 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.194@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:36:46 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.194@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:36:46.364581 16328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:46.358260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:34777 (local address 127.6.246.194:58443)
0925 22:36:46.358821 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:46.358856 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:46.358993 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:46.359287 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:46.359289 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:46.359512 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:46.359640 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:46.359646 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:46.360411 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:46.360415 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:46.361088 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:46.361098 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:46.361607 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:46.361611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:46.361690 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:46.362284 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:46.362296 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:46.364381 (+ 2085us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.246.254:34777: Server oryx/127.6.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":132,"threads_started":1}
W20260925 22:36:46.364832 16327 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:36:46.358279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.194:58443 (local address 127.6.246.254:34777)
0925 22:36:46.358434 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:46.358442 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:46.358938 (+ 496us) server_negotiation.cc:407] Connection header received
0925 22:36:46.359104 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:46.359107 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:46.359173 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:46.359250 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:46.359734 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:46.360297 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:46.361218 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:46.361457 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:46.361551 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:46.364578 (+ 3027us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.246.194:58443: BlockingRecv error: recv got EOF from 127.6.246.194:58443 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 25 22:36:46 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.195@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:36:46 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.195@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:36:46.550309 16329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:46.543933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:34777 (local address 127.6.246.195:49635)
0925 22:36:46.544346 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:46.544360 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:46.544531 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:46.544820 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:46.544823 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:46.545091 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:46.545203 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:46.545210 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:46.546197 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:46.546202 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:46.546928 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:46.546939 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:46.547391 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:46.547395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:46.547470 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:46.548038 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:46.548049 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:46.550161 (+ 2112us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.246.254:34777: Server oryx/127.6.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":138,"threads_started":1}
W20260925 22:36:46.550422 16327 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:36:46.544022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.195:49635 (local address 127.6.246.254:34777)
0925 22:36:46.544232 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:46.544237 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:46.544442 (+ 205us) server_negotiation.cc:407] Connection header received
0925 22:36:46.544646 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:46.544649 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:46.544711 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:46.544794 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:46.545337 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:46.546093 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:46.547039 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:46.547283 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:46.547391 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:46.550298 (+ 2907us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.246.195:49635: BlockingRecv error: recv got EOF from 127.6.246.195:49635 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 25 22:36:47 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.193@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:36:47 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.193@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:36:47.185967 16330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:47.179032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:34777 (local address 127.6.246.193:53483)
0925 22:36:47.179410 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:47.179427 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:47.179574 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:47.179856 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:47.179858 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:47.180119 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:47.180248 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:47.180255 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:47.181174 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:47.181178 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:47.182206 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:47.182221 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:47.182759 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:47.182762 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:47.182831 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:47.183559 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:47.183573 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:47.185765 (+ 2192us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.246.254:34777: Server oryx/127.6.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":109,"threads_started":1}
W20260925 22:36:47.186036 16331 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:36:47.179207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.193:53483 (local address 127.6.246.254:34777)
0925 22:36:47.179462 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:47.179465 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:47.179497 (+ 32us) server_negotiation.cc:407] Connection header received
0925 22:36:47.179670 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:47.179672 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:47.179725 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:47.179807 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:47.180354 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:47.181027 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:47.182334 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:47.182650 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:47.182736 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:47.185901 (+ 3165us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.246.193:53483: BlockingRecv error: recv got EOF from 127.6.246.193:53483 (error 108)
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":69,"threads_started":1}
Sep 25 22:36:47 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.194@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:36:47 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.194@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:36:47.372186 16332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:47.365534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:34777 (local address 127.6.246.194:55909)
0925 22:36:47.365839 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:47.365853 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:47.365974 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:47.366267 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:47.366270 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:47.366495 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:47.366622 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:47.366628 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:47.367749 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:47.367756 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:47.368682 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:47.368696 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:47.369268 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:47.369271 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:47.369340 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:47.369916 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:47.369928 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:47.372030 (+ 2102us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.246.254:34777: Server oryx/127.6.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":104,"threads_started":1}
W20260925 22:36:47.372452 16331 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:36:47.365598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.194:55909 (local address 127.6.246.254:34777)
0925 22:36:47.365779 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:47.365782 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:47.365933 (+ 151us) server_negotiation.cc:407] Connection header received
0925 22:36:47.366085 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:47.366087 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:47.366142 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:47.366223 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:47.366744 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:47.367626 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:47.368802 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:47.369152 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:47.369252 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:47.372199 (+ 2947us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.246.194:55909: BlockingRecv error: recv got EOF from 127.6.246.194:55909 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 25 22:36:47 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.195@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:36:47 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.195@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:36:47.562106 16333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:47.551330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:34777 (local address 127.6.246.195:33047)
0925 22:36:47.551661 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:47.551675 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:47.551835 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:47.552118 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:47.552121 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:47.552379 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:47.552549 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:47.552561 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:47.553412 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:47.553419 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:47.554048 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:47.554057 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:47.554526 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:47.554530 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:47.554613 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:47.559556 (+ 4943us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:47.559570 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:47.561949 (+ 2379us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.246.254:34777: Server oryx/127.6.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":115,"threads_started":1}
W20260925 22:36:47.562342 16331 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:36:47.551389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.195:33047 (local address 127.6.246.254:34777)
0925 22:36:47.551581 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:47.551587 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:47.551760 (+ 173us) server_negotiation.cc:407] Connection header received
0925 22:36:47.551943 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:47.551946 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:47.552007 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:47.552097 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:47.552729 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:47.553300 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:47.554171 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:47.554409 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:47.554519 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:47.562193 (+ 7674us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.246.195:33047: BlockingRecv error: recv got EOF from 127.6.246.195:33047 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 25 22:36:48 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.193@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:36:48 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.193@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:36:48.193236 16334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:48.186990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:34777 (local address 127.6.246.193:42029)
0925 22:36:48.187321 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:48.187337 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:48.187455 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:48.187720 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:48.187722 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:48.187938 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:48.188057 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:48.188064 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:48.188858 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:48.188860 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:48.189532 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:48.189546 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:48.190158 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:48.190161 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:48.190247 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:48.190913 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:48.190926 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:48.193096 (+ 2170us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.246.254:34777: Server oryx/127.6.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":119,"threads_started":1}
W20260925 22:36:48.193495 16335 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:36:48.187065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.193:42029 (local address 127.6.246.254:34777)
0925 22:36:48.187329 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:48.187334 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:48.187415 (+ 81us) server_negotiation.cc:407] Connection header received
0925 22:36:48.187538 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:48.187540 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:48.187592 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:48.187678 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:48.188162 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:48.188772 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:48.189664 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:48.190025 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:48.190162 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:48.193248 (+ 3086us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.246.193:42029: BlockingRecv error: recv got EOF from 127.6.246.193:42029 (error 108)
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":72,"threads_started":1}
Sep 25 22:36:48 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.194@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:36:48 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.194@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:36:48.380298 16336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:48.373157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:34777 (local address 127.6.246.194:44863)
0925 22:36:48.373482 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:48.373497 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:48.373626 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:48.373987 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:48.373991 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:48.374325 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:48.374485 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:48.374492 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:48.375651 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:48.375657 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:48.376350 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:48.376360 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:48.376980 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:48.376984 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:48.377074 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:48.377728 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:48.377740 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:48.380084 (+ 2344us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.246.254:34777: Server oryx/127.6.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":109,"threads_started":1}
W20260925 22:36:48.380350 16335 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:36:48.373283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.194:44863 (local address 127.6.246.254:34777)
0925 22:36:48.373497 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:48.373502 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:48.373581 (+ 79us) server_negotiation.cc:407] Connection header received
0925 22:36:48.373720 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:48.373723 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:48.373827 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:48.373926 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:48.374636 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:48.375520 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:48.376507 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:48.376853 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:48.376967 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:48.380233 (+ 3266us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.246.194:44863: BlockingRecv error: recv got EOF from 127.6.246.194:44863 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Sep 25 22:36:48 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.195@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:36:48 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.195@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:36:48.569159 16337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:48.562967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:34777 (local address 127.6.246.195:57911)
0925 22:36:48.563265 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:48.563279 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:48.563397 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:48.563678 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:48.563681 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:48.563955 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:48.564144 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:48.564158 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:48.565281 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:48.565284 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:48.565895 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:48.565902 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:48.566291 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:48.566293 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:48.566369 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:48.566895 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:48.566907 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:48.568982 (+ 2075us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.246.254:34777: Server oryx/127.6.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":115,"threads_started":1}
W20260925 22:36:48.569213 16335 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:36:48.563014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.195:57911 (local address 127.6.246.254:34777)
0925 22:36:48.563167 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:48.563173 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:48.563352 (+ 179us) server_negotiation.cc:407] Connection header received
0925 22:36:48.563489 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:48.563491 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:48.563565 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:48.563628 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:48.564268 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:48.565186 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:48.565988 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:48.566203 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:48.566278 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:48.569111 (+ 2833us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.246.195:57911: BlockingRecv error: recv got EOF from 127.6.246.195:57911 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 25 22:36:49 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.193@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:36:49 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.193@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:36:49.200590 16338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:49.194300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:34777 (local address 127.6.246.193:58315)
0925 22:36:49.194676 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:49.194692 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:49.194802 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:49.195092 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:49.195094 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:49.195326 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:49.195449 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:49.195454 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:49.196248 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:49.196250 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:49.196873 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:49.196881 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:49.197338 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:49.197340 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:49.197405 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:49.198018 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:49.198030 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:49.200342 (+ 2312us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.246.254:34777: Server oryx/127.6.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":110,"threads_started":1}
W20260925 22:36:49.200965 16339 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:36:49.194449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.193:58315 (local address 127.6.246.254:34777)
0925 22:36:49.194771 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:49.194775 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:49.194791 (+ 16us) server_negotiation.cc:407] Connection header received
0925 22:36:49.194908 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:49.194910 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:49.194962 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:49.195049 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:49.195557 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:49.196146 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:49.196977 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:49.197226 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:49.197316 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:49.200614 (+ 3298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.246.193:58315: BlockingRecv error: recv got EOF from 127.6.246.193:58315 (error 108)
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":105,"threads_started":1}
Sep 25 22:36:49 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.194@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:36:49 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.194@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:36:49.387105 16340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:49.381340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:34777 (local address 127.6.246.194:56019)
0925 22:36:49.381647 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:49.381663 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:49.381773 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:49.382065 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:49.382067 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:49.382298 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:49.382449 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:49.382457 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:49.383311 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:49.383314 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:49.383944 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:49.383952 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:49.384366 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:49.384369 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:49.384429 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:49.385021 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:49.385032 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:49.386944 (+ 1912us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.246.254:34777: Server oryx/127.6.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":106,"threads_started":1}
W20260925 22:36:49.387274 16339 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:36:49.381410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.194:56019 (local address 127.6.246.254:34777)
0925 22:36:49.381606 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:49.381610 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:49.381768 (+ 158us) server_negotiation.cc:407] Connection header received
0925 22:36:49.381851 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:49.381853 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:49.381907 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:49.381999 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:49.382603 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:49.383209 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:49.384047 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:49.384270 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:49.384349 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:49.387099 (+ 2750us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.246.194:56019: BlockingRecv error: recv got EOF from 127.6.246.194:56019 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 25 22:36:49 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.195@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:36:49 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.195@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:36:49.576290 16341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:49.570128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:34777 (local address 127.6.246.195:59253)
0925 22:36:49.570493 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:49.570516 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:49.570655 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:49.570941 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:49.570944 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:49.571213 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:49.571323 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:49.571329 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:49.572147 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:49.572154 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:49.572836 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:49.572844 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:49.573323 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:49.573326 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:49.573404 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:49.573986 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:49.573998 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:49.576117 (+ 2119us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.246.254:34777: Server oryx/127.6.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":110,"threads_started":1}
W20260925 22:36:49.576625 16339 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:36:49.570278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.195:59253 (local address 127.6.246.254:34777)
0925 22:36:49.570451 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:49.570456 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:49.570593 (+ 137us) server_negotiation.cc:407] Connection header received
0925 22:36:49.570767 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:49.570770 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:49.570836 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:49.570920 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:49.571444 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:49.572030 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:49.572958 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:49.573210 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:49.573322 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:49.576308 (+ 2986us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.246.195:59253: BlockingRecv error: recv got EOF from 127.6.246.195:59253 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 25 22:36:50 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.193@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:36:50 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.193@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:36:50.207609 16342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:50.201453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:34777 (local address 127.6.246.193:49519)
0925 22:36:50.201804 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:50.201818 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:50.201931 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:50.202220 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:50.202222 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:50.202445 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:50.202577 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:50.202582 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:50.203422 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:50.203425 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:50.204051 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:50.204061 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:50.204598 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:50.204601 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:50.204671 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:50.205275 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:50.205288 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:50.207465 (+ 2177us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.246.254:34777: Server oryx/127.6.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":99,"threads_started":1}
W20260925 22:36:50.207930 16343 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:36:50.201541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.193:49519 (local address 127.6.246.254:34777)
0925 22:36:50.201858 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:50.201861 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:50.201895 (+ 34us) server_negotiation.cc:407] Connection header received
0925 22:36:50.202030 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:50.202032 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:50.202085 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:50.202177 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:50.202692 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:50.203308 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:50.204177 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:50.204424 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:50.204587 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:50.207629 (+ 3042us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.246.193:49519: BlockingRecv error: recv got EOF from 127.6.246.193:49519 (error 108)
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":102,"threads_started":1}
Sep 25 22:36:50 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.194@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:36:50 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.194@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:36:50.394237 16344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:50.388097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:34777 (local address 127.6.246.194:58483)
0925 22:36:50.388431 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:50.388446 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:50.388589 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:50.388856 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:50.388859 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:50.389077 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:50.389219 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:50.389225 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:50.390050 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:50.390053 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:50.390667 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:50.390675 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:50.391154 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:50.391157 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:50.391223 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:50.391862 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:50.391875 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:50.394067 (+ 2192us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.246.254:34777: Server oryx/127.6.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":128,"threads_started":1}
W20260925 22:36:50.394410 16343 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:36:50.388157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.194:58483 (local address 127.6.246.254:34777)
0925 22:36:50.388324 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:50.388328 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:50.388574 (+ 246us) server_negotiation.cc:407] Connection header received
0925 22:36:50.388679 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:50.388682 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:50.388745 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:50.388812 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:50.389356 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:50.389948 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:50.390785 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:50.391019 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:50.391115 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:50.394233 (+ 3118us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.246.194:58483: BlockingRecv error: recv got EOF from 127.6.246.194:58483 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 25 22:36:50 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.195@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:36:50 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.195@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:36:50.584199 16345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:50.577321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:34777 (local address 127.6.246.195:50415)
0925 22:36:50.577708 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:50.577732 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:50.577890 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:50.578258 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:50.578261 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:50.578523 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:50.578652 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:50.578660 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:50.579535 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:50.579540 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:50.580161 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:50.580170 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:50.580783 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:50.580787 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:50.580904 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:50.581623 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:50.581644 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:50.583981 (+ 2337us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.246.254:34777: Server oryx/127.6.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":138,"threads_started":1}
W20260925 22:36:50.584515 16343 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:36:50.577392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.195:50415 (local address 127.6.246.254:34777)
0925 22:36:50.577629 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:50.577636 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:50.577821 (+ 185us) server_negotiation.cc:407] Connection header received
0925 22:36:50.578012 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:50.578016 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:50.578121 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:50.578232 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:50.578784 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:50.579408 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:50.580290 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:50.580659 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:50.580784 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:50.584188 (+ 3404us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.246.195:50415: BlockingRecv error: recv got EOF from 127.6.246.195:50415 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 25 22:36:51 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.193@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:36:51 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.193@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:36:51.215119 16346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:51.208608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:34777 (local address 127.6.246.193:32815)
0925 22:36:51.209031 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:51.209047 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:51.209209 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:51.209532 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:51.209535 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:51.209767 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:51.209894 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:51.209901 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:51.210704 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:51.210707 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:51.211475 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:51.211486 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:51.211996 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:51.212000 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:51.212091 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:51.212832 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:51.212845 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:51.214949 (+ 2104us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.246.254:34777: Server oryx/127.6.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":129,"threads_started":1}
W20260925 22:36:51.215315 16347 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:36:51.208748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.193:32815 (local address 127.6.246.254:34777)
0925 22:36:51.209070 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:51.209074 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:51.209123 (+ 49us) server_negotiation.cc:407] Connection header received
0925 22:36:51.209310 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:51.209312 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:51.209366 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:51.209455 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:51.210008 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:51.210606 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:51.211609 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:51.211860 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:51.211986 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:51.215102 (+ 3116us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.246.193:32815: BlockingRecv error: recv got EOF from 127.6.246.193:32815 (error 108)
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":110,"threads_started":1}
Sep 25 22:36:51 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.194@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:36:51 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.194@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:36:51.401657 16348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:51.395337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:34777 (local address 127.6.246.194:53655)
0925 22:36:51.395707 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:51.395721 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:51.395850 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:51.396130 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:51.396132 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:51.396358 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:51.396520 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:51.396527 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:51.397355 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:51.397358 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:51.398052 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:51.398061 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:51.398672 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:51.398676 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:51.398755 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:51.399427 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:51.399439 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:51.401454 (+ 2015us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.246.254:34777: Server oryx/127.6.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":143,"threads_started":1}
W20260925 22:36:51.401732 16347 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:36:51.395485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.194:53655 (local address 127.6.246.254:34777)
0925 22:36:51.395666 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:51.395670 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:51.395812 (+ 142us) server_negotiation.cc:407] Connection header received
0925 22:36:51.395952 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:51.395954 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:51.396007 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:51.396079 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:51.396622 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:51.397242 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:51.398208 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:51.398549 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:51.398667 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:51.401612 (+ 2945us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.246.194:53655: BlockingRecv error: recv got EOF from 127.6.246.194:53655 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 25 22:36:51 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.195@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:36:51 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.195@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:36:51.592543 16349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:51.585320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:34777 (local address 127.6.246.195:37285)
0925 22:36:51.585642 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:51.585656 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:51.585822 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:51.586208 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:51.586211 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:51.586453 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:51.586605 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:51.586614 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:51.587446 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:51.587449 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:51.588048 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:51.588055 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:51.588511 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:51.588514 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:51.588581 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:51.589404 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:51.589439 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:51.592291 (+ 2852us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.246.254:34777: Server oryx/127.6.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":114,"threads_started":1}
W20260925 22:36:51.592676 16347 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:36:51.585375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.195:37285 (local address 127.6.246.254:34777)
0925 22:36:51.585539 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:51.585544 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:51.585795 (+ 251us) server_negotiation.cc:407] Connection header received
0925 22:36:51.585959 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:51.585962 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:51.586034 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:51.586139 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:51.586729 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:51.587309 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:51.588159 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:51.588375 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:51.588516 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:51.592515 (+ 3999us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.246.195:37285: BlockingRecv error: recv got EOF from 127.6.246.195:37285 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 25 22:36:52 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.193@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:36:52 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.193@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:36:52.222815 16350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:52.216144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:34777 (local address 127.6.246.193:51715)
0925 22:36:52.216545 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:52.216564 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:52.216706 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:52.217027 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:52.217031 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:52.217335 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:52.217458 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:52.217465 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:52.218326 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:52.218330 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:52.218931 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:52.218939 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:52.219416 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:52.219420 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:52.219513 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:52.220130 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:52.220143 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:52.222664 (+ 2521us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.246.254:34777: Server oryx/127.6.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":136,"threads_started":1}
W20260925 22:36:52.222981 16351 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:36:52.216179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.193:51715 (local address 127.6.246.254:34777)
0925 22:36:52.216545 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:52.216551 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:52.216655 (+ 104us) server_negotiation.cc:407] Connection header received
0925 22:36:52.216825 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:52.216829 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:52.216890 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:52.216995 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:52.217589 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:52.218208 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:52.219054 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:52.219298 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:52.219410 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:52.222829 (+ 3419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.246.193:51715: BlockingRecv error: recv got EOF from 127.6.246.193:51715 (error 108)
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":108,"threads_started":1}
Sep 25 22:36:52 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.194@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:36:52 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.194@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:36:52.409183 16352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:52.402772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:34777 (local address 127.6.246.194:56043)
0925 22:36:52.403370 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:52.403385 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:52.403547 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:52.403857 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:52.403860 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:52.404136 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:52.404249 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:52.404256 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:52.405184 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:52.405189 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:52.405802 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:52.405810 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:52.406281 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:52.406285 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:52.406368 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:52.406942 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:52.406953 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:52.408973 (+ 2020us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.246.254:34777: Server oryx/127.6.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":153,"threads_started":1}
W20260925 22:36:52.409233 16351 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:36:52.403002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.194:56043 (local address 127.6.246.254:34777)
0925 22:36:52.403191 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:52.403197 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:52.403482 (+ 285us) server_negotiation.cc:407] Connection header received
0925 22:36:52.403674 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:52.403678 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:52.403736 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:52.403832 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:52.404387 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:52.405064 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:52.405923 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:52.406164 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:52.406280 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:52.409101 (+ 2821us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.246.194:56043: BlockingRecv error: recv got EOF from 127.6.246.194:56043 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 25 22:36:52 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.195@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:36:52 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.195@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:36:52.604672 16353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:52.593500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:34777 (local address 127.6.246.195:33967)
0925 22:36:52.593842 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:52.593856 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:52.593986 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:52.594263 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:52.594266 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:52.594482 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:52.594596 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:52.594602 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:52.595437 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:52.595439 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:52.596026 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:52.596034 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:52.596680 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:52.596684 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:52.596752 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:52.601965 (+ 5213us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:52.601978 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:52.604414 (+ 2436us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.246.254:34777: Server oryx/127.6.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":125,"threads_started":1}
W20260925 22:36:52.604753 16351 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:36:52.593583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.195:33967 (local address 127.6.246.254:34777)
0925 22:36:52.593730 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:52.593734 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:52.593945 (+ 211us) server_negotiation.cc:407] Connection header received
0925 22:36:52.594083 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:52.594085 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:52.594149 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:52.594218 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:52.594728 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:52.595336 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:52.596160 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:52.596403 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:52.596524 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:52.604616 (+ 8092us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.246.195:33967: BlockingRecv error: recv got EOF from 127.6.246.195:33967 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 25 22:36:53 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.193@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:36:53 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.193@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:36:53.230341 16354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:53.224034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:34777 (local address 127.6.246.193:58357)
0925 22:36:53.224429 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:53.224445 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:53.224625 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:53.224906 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:53.224909 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:53.225156 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:53.225275 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:53.225280 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:53.226066 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:53.226068 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:53.226660 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:53.226667 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:53.227165 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:53.227168 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:53.227236 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:53.227802 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:53.227813 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:53.230147 (+ 2334us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.246.254:34777: Server oryx/127.6.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":139,"threads_started":1}
W20260925 22:36:53.230526 16355 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:36:53.224028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.193:58357 (local address 127.6.246.254:34777)
0925 22:36:53.224472 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:53.224476 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:53.224579 (+ 103us) server_negotiation.cc:407] Connection header received
0925 22:36:53.224717 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:53.224719 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:53.224771 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:53.224859 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:53.225381 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:53.225972 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:53.226804 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:53.227040 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:53.227124 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:53.230337 (+ 3213us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.246.193:58357: BlockingRecv error: recv got EOF from 127.6.246.193:58357 (error 108)
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":130,"threads_started":1}
Sep 25 22:36:53 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.194@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:36:53 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.194@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:36:53.416178 16356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:53.410204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:34777 (local address 127.6.246.194:36359)
0925 22:36:53.410553 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:53.410567 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:53.410720 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:53.411006 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:53.411009 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:53.411259 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:53.411367 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:53.411373 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:53.412153 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:53.412159 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:53.412769 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:53.412778 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:53.413370 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:53.413374 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:53.413468 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:53.414092 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:53.414105 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:53.416015 (+ 1910us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.246.254:34777: Server oryx/127.6.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":112,"threads_started":1}
W20260925 22:36:53.416253 16355 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:36:53.410253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.194:36359 (local address 127.6.246.254:34777)
0925 22:36:53.410437 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:53.410443 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:53.410654 (+ 211us) server_negotiation.cc:407] Connection header received
0925 22:36:53.410821 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:53.410825 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:53.410883 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:53.410975 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:53.411482 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:53.412036 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:53.412886 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:53.413231 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:53.413359 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:53.416153 (+ 2794us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.246.194:36359: BlockingRecv error: recv got EOF from 127.6.246.194:36359 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 25 22:36:53 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.195@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:36:53 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.195@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:36:53.611647 16357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:53.605710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:34777 (local address 127.6.246.195:46793)
0925 22:36:53.606012 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:53.606026 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:53.606161 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:53.606426 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:53.606428 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:53.606640 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:53.606770 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:53.606776 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:53.607546 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:53.607548 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:53.608152 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:53.608160 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:53.608586 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:53.608588 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:53.608652 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:53.609187 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:53.609198 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:53.611489 (+ 2291us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.246.254:34777: Server oryx/127.6.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":97,"threads_started":1}
W20260925 22:36:53.611723 16355 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:36:53.605814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.195:46793 (local address 127.6.246.254:34777)
0925 22:36:53.606007 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:53.606012 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:53.606117 (+ 105us) server_negotiation.cc:407] Connection header received
0925 22:36:53.606245 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:53.606247 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:53.606301 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:53.606391 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:53.606869 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:53.607458 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:53.608260 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:53.608498 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:53.608571 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:53.611619 (+ 3048us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.246.195:46793: BlockingRecv error: recv got EOF from 127.6.246.195:46793 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 25 22:36:54 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.193@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:36:54 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.193@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:36:54.238102 16358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:54.231701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:34777 (local address 127.6.246.193:43597)
0925 22:36:54.232146 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:54.232160 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:54.232266 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:54.232581 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:54.232584 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:54.232852 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:54.232970 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:54.232978 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:54.233746 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:54.233748 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:54.234374 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:54.234381 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:54.234869 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:54.234871 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:54.234944 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:54.235530 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:54.235542 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:54.237883 (+ 2341us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.246.254:34777: Server oryx/127.6.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":219,"threads_started":1}
W20260925 22:36:54.238158 16359 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:36:54.231873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.193:43597 (local address 127.6.246.254:34777)
0925 22:36:54.232152 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:54.232156 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:54.232233 (+ 77us) server_negotiation.cc:407] Connection header received
0925 22:36:54.232349 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:54.232351 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:54.232400 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:54.232568 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:54.233073 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:54.233656 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:54.234505 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:54.234755 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:54.234857 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:54.238019 (+ 3162us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.246.193:43597: BlockingRecv error: recv got EOF from 127.6.246.193:43597 (error 108)
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":121,"threads_started":1}
Sep 25 22:36:54 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.194@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:36:54 dist-test-slave-wjdr krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.246.194@KRBTEST.COM for oryx/127.6.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:36:54.423710 16360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:54.417508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:34777 (local address 127.6.246.194:45043)
0925 22:36:54.417878 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:54.417894 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:54.418014 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:54.418254 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:54.418256 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:54.418475 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:54.418587 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:54.418604 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:54.419396 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:54.419399 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:54.420005 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:54.420015 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:54.420448 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:54.420482 (+ 34us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:54.420546 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:54.421105 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:54.421116 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:54.423538 (+ 2422us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.246.254:34777: Server oryx/127.6.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":142,"threads_started":1}
W20260925 22:36:54.423873 16359 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:36:54.417584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.194:45043 (local address 127.6.246.254:34777)
0925 22:36:54.417766 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:54.417771 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:54.417972 (+ 201us) server_negotiation.cc:407] Connection header received
0925 22:36:54.418090 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:54.418092 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:54.418143 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:54.418217 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:54.418697 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:54.419281 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:54.420128 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:54.420352 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:54.420432 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:54.423730 (+ 3298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.246.194:45043: BlockingRecv error: recv got EOF from 127.6.246.194:45043 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 22:36:54.491168 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 15884
I20260925 22:36:54.498020 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 16036
I20260925 22:36:54.504335 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 16168
I20260925 22:36:54.509986 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 15814
2026-09-25T22:36:54Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17984 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-25T22:36:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:36:54Z Disabled control of system clock
I20260925 22:36:54.542850 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790375752797504-7131-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790375752797504-7131-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790375752797504-7131-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:33765
--webserver_interface=127.6.246.254
--webserver_port=0
--builtin_ntp_servers=127.6.246.212:45181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.246.254:33765
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 22:36:54.667270 16364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:36:54.667548 16364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:36:54.667587 16364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:36:54.671691 16364 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:36:54.671792 16364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:36:54.671818 16364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:36:54.671851 16364 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:36:54.671905 16364 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:36:54.676877 16364 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:45181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790375752797504-7131-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790375752797504-7131-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.246.254:33765
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:33765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.246.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16364
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790375752797504-7131-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:36:54.678263 16364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:36:54.679962 16364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:36:54.689445 16369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:54.689445 16372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:54.689456 16370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:54.689744 16364 server_base.cc:1034] running on GCE node
I20260925 22:36:54.690415 16364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:36:54.691660 16364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:36:54.692834 16364 hybrid_clock.cc:648] HybridClock initialized: now 1790375814692815 us; error 48 us; skew 500 ppm
I20260925 22:36:54.694950 16364 webserver.cc:460] Webserver started at http://127.6.246.254:37035/ using document root <none> and password file <none>
I20260925 22:36:54.695631 16364 fs_manager.cc:359] Metadata directory not provided
I20260925 22:36:54.695720 16364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:36:54.695972 16364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:36:54.698536 16364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790375752797504-7131-0/minicluster-data/master-0/data/instance:
uuid: "173fd3a3453245d9838056aecf40e7ea"
format_stamp: "Formatted at 2026-09-25 22:36:54 on dist-test-slave-wjdr"
server_key: "d4dd25dcaeb9120bcec6fde4d072bacd"
server_key_iv: "49428f20d4e674d97130eb89f57e4126"
server_key_version: "encryptionkey@0"
I20260925 22:36:54.699185 16364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790375752797504-7131-0/minicluster-data/master-0/wal/instance:
uuid: "173fd3a3453245d9838056aecf40e7ea"
format_stamp: "Formatted at 2026-09-25 22:36:54 on dist-test-slave-wjdr"
server_key: "d4dd25dcaeb9120bcec6fde4d072bacd"
server_key_iv: "49428f20d4e674d97130eb89f57e4126"
server_key_version: "encryptionkey@0"
I20260925 22:36:54.703356 16364 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260925 22:36:54.705987 16378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:54.707319 16364 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 22:36:54.707454 16364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790375752797504-7131-0/minicluster-data/master-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790375752797504-7131-0/minicluster-data/master-0/wal
uuid: "173fd3a3453245d9838056aecf40e7ea"
format_stamp: "Formatted at 2026-09-25 22:36:54 on dist-test-slave-wjdr"
server_key: "d4dd25dcaeb9120bcec6fde4d072bacd"
server_key_iv: "49428f20d4e674d97130eb89f57e4126"
server_key_version: "encryptionkey@0"
I20260925 22:36:54.707571 16364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790375752797504-7131-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790375752797504-7131-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790375752797504-7131-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:36:54.734896 16364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:36:54.735790 16364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:36:54.736012 16364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:36:54.744181 16364 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.254:33765
I20260925 22:36:54.744192 16430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.254:33765 every 8 connection(s)
I20260925 22:36:54.745309 16364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
I20260925 22:36:54.748900 16431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:36:54.749446 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 16364
I20260925 22:36:54.749595 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790375752797504-7131-0/minicluster-data/master-0/wal/instance
I20260925 22:36:54.749939 7131 external_mini_cluster.cc:1428] Setting key fef70ff684933821e4ecd7cefa5890e7
I20260925 22:36:54.757025 16431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 173fd3a3453245d9838056aecf40e7ea: Bootstrap starting.
I20260925 22:36:54.760226 16431 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 173fd3a3453245d9838056aecf40e7ea: Neither blocks nor log segments found. Creating new log.
I20260925 22:36:54.761334 16431 log.cc:824] T 00000000000000000000000000000000 P 173fd3a3453245d9838056aecf40e7ea: Log is configured to *not* fsync() on all Append() calls
I20260925 22:36:54.761998 16434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:54.751666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35382 (local address 127.6.246.254:33765)
0925 22:36:54.752484 (+ 818us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:54.752507 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:54.752562 (+ 55us) server_negotiation.cc:407] Connection header received
0925 22:36:54.753551 (+ 989us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:54.753579 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:54.754204 (+ 625us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:54.754623 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:54.755142 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:54.756731 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:54.757700 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:54.758405 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:54.758597 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:54.759460 (+ 863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:54.759467 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:54.759486 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:36:54.760056 (+ 570us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:54.760140 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:36:54.760175 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:54.760281 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:54.760573 (+ 292us) server_negotiation.cc:299] Negotiation successful
0925 22:36:54.760907 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":123,"threads_started":1}
I20260925 22:36:54.763804 16431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 173fd3a3453245d9838056aecf40e7ea: No bootstrap required, opened a new log
I20260925 22:36:54.768077 16431 raft_consensus.cc:348] T 00000000000000000000000000000000 P 173fd3a3453245d9838056aecf40e7ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "173fd3a3453245d9838056aecf40e7ea" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 33765 } }
I20260925 22:36:54.768370 16431 raft_consensus.cc:374] T 00000000000000000000000000000000 P 173fd3a3453245d9838056aecf40e7ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:36:54.768445 16431 raft_consensus.cc:729] T 00000000000000000000000000000000 P 173fd3a3453245d9838056aecf40e7ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 173fd3a3453245d9838056aecf40e7ea, State: Initialized, Role: FOLLOWER
I20260925 22:36:54.768971 16431 consensus_queue.cc:260] T 00000000000000000000000000000000 P 173fd3a3453245d9838056aecf40e7ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "173fd3a3453245d9838056aecf40e7ea" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 33765 } }
I20260925 22:36:54.769141 16431 raft_consensus.cc:388] T 00000000000000000000000000000000 P 173fd3a3453245d9838056aecf40e7ea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:36:54.769227 16431 raft_consensus.cc:482] T 00000000000000000000000000000000 P 173fd3a3453245d9838056aecf40e7ea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:36:54.769353 16431 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 173fd3a3453245d9838056aecf40e7ea [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:36:54.770421 16431 raft_consensus.cc:504] T 00000000000000000000000000000000 P 173fd3a3453245d9838056aecf40e7ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "173fd3a3453245d9838056aecf40e7ea" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 33765 } }
I20260925 22:36:54.770805 16431 leader_election.cc:302] T 00000000000000000000000000000000 P 173fd3a3453245d9838056aecf40e7ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 173fd3a3453245d9838056aecf40e7ea; no voters:
I20260925 22:36:54.771168 16431 leader_election.cc:288] T 00000000000000000000000000000000 P 173fd3a3453245d9838056aecf40e7ea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:36:54.771409 16436 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 173fd3a3453245d9838056aecf40e7ea [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:36:54.771731 16436 raft_consensus.cc:686] T 00000000000000000000000000000000 P 173fd3a3453245d9838056aecf40e7ea [term 1 LEADER]: Becoming Leader. State: Replica: 173fd3a3453245d9838056aecf40e7ea, State: Running, Role: LEADER
I20260925 22:36:54.772068 16436 consensus_queue.cc:237] T 00000000000000000000000000000000 P 173fd3a3453245d9838056aecf40e7ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "173fd3a3453245d9838056aecf40e7ea" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 33765 } }
I20260925 22:36:54.772686 16431 sys_catalog.cc:564] T 00000000000000000000000000000000 P 173fd3a3453245d9838056aecf40e7ea [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:36:54.774211 16437 sys_catalog.cc:455] T 00000000000000000000000000000000 P 173fd3a3453245d9838056aecf40e7ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "173fd3a3453245d9838056aecf40e7ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "173fd3a3453245d9838056aecf40e7ea" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 33765 } } }
I20260925 22:36:54.774358 16437 sys_catalog.cc:458] T 00000000000000000000000000000000 P 173fd3a3453245d9838056aecf40e7ea [sys.catalog]: This master's current role is: LEADER
I20260925 22:36:54.774199 16438 sys_catalog.cc:455] T 00000000000000000000000000000000 P 173fd3a3453245d9838056aecf40e7ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 173fd3a3453245d9838056aecf40e7ea. Latest consensus state: current_term: 1 leader_uuid: "173fd3a3453245d9838056aecf40e7ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "173fd3a3453245d9838056aecf40e7ea" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 33765 } } }
I20260925 22:36:54.774441 16438 sys_catalog.cc:458] T 00000000000000000000000000000000 P 173fd3a3453245d9838056aecf40e7ea [sys.catalog]: This master's current role is: LEADER
I20260925 22:36:54.774778 16445 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:36:54.778060 16445 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:36:54.784606 16445 catalog_manager.cc:1347] Generated new cluster ID: 0d80fea50cad4102b58058264422e6c1
I20260925 22:36:54.784677 16445 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:36:54.794219 16445 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:36:54.795228 16445 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:36:54.811403 16445 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 173fd3a3453245d9838056aecf40e7ea: Generated new TSK 0
I20260925 22:36:54.812366 16445 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:36:54.821591 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.193:0
--local_ip_for_outbound_sockets=127.6.246.193
--webserver_interface=127.6.246.193
--webserver_port=0
--tserver_master_addrs=127.6.246.254:33765
--builtin_ntp_servers=127.6.246.212:45181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260925 22:36:54.949409 16455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:36:54.949680 16455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:36:54.949719 16455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:36:54.953859 16455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:36:54.953944 16455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:36:54.954036 16455 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.193
I20260925 22:36:54.958837 16455 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:45181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.246.193
--webserver_port=0
--tserver_master_addrs=127.6.246.254:33765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.193
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:36:54.960035 16455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:36:54.961665 16455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:36:54.971746 16463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:54.971719 16461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:54.971715 16460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:54.971815 16455 server_base.cc:1034] running on GCE node
I20260925 22:36:54.972389 16455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:36:54.973057 16455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:36:54.974222 16455 hybrid_clock.cc:648] HybridClock initialized: now 1790375814974211 us; error 32 us; skew 500 ppm
I20260925 22:36:54.976065 16455 webserver.cc:460] Webserver started at http://127.6.246.193:34715/ using document root <none> and password file <none>
I20260925 22:36:54.976729 16455 fs_manager.cc:359] Metadata directory not provided
I20260925 22:36:54.976792 16455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:36:54.977013 16455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:36:54.979086 16455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-0/data/instance:
uuid: "56733672a82e42b4855a4624237c68ee"
format_stamp: "Formatted at 2026-09-25 22:36:54 on dist-test-slave-wjdr"
server_key: "3f27af8da701f4e006045b0d32071441"
server_key_iv: "834525dc78170b368b3a30929e9c5b84"
server_key_version: "encryptionkey@0"
I20260925 22:36:54.979656 16455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-0/wal/instance:
uuid: "56733672a82e42b4855a4624237c68ee"
format_stamp: "Formatted at 2026-09-25 22:36:54 on dist-test-slave-wjdr"
server_key: "3f27af8da701f4e006045b0d32071441"
server_key_iv: "834525dc78170b368b3a30929e9c5b84"
server_key_version: "encryptionkey@0"
I20260925 22:36:54.983386 16455 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 22:36:54.985975 16469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:54.987246 16455 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 22:36:54.987385 16455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-0/wal
uuid: "56733672a82e42b4855a4624237c68ee"
format_stamp: "Formatted at 2026-09-25 22:36:54 on dist-test-slave-wjdr"
server_key: "3f27af8da701f4e006045b0d32071441"
server_key_iv: "834525dc78170b368b3a30929e9c5b84"
server_key_version: "encryptionkey@0"
I20260925 22:36:54.987514 16455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:36:55.004034 16455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:36:55.004824 16455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:36:55.005038 16455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:36:55.005630 16455 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:36:55.006592 16455 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:36:55.006660 16455 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:55.006729 16455 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:36:55.006774 16455 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:55.016039 16455 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.193:46113
I20260925 22:36:55.016084 16582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.193:46113 every 8 connection(s)
I20260925 22:36:55.017010 16455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
I20260925 22:36:55.019217 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 16455
I20260925 22:36:55.019348 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-0/wal/instance
I20260925 22:36:55.019671 7131 external_mini_cluster.cc:1428] Setting key 150d85a78d2bdeca2c2e7127182d3e6b
I20260925 22:36:55.021911 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.194:0
--local_ip_for_outbound_sockets=127.6.246.194
--webserver_interface=127.6.246.194
--webserver_port=0
--tserver_master_addrs=127.6.246.254:33765
--builtin_ntp_servers=127.6.246.212:45181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 22:36:55.026633 16434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:55.018900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.193:57477 (local address 127.6.246.254:33765)
0925 22:36:55.019074 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:55.019080 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:55.020278 (+ 1198us) server_negotiation.cc:407] Connection header received
0925 22:36:55.021872 (+ 1594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:55.021875 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:55.021933 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:55.022015 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:55.022896 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:55.023447 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:55.024610 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:55.024839 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:55.024954 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:55.026136 (+ 1182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:55.026140 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:55.026143 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:36:55.026160 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:55.026178 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:36:55.026186 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:55.026261 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:55.026420 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 22:36:55.026483 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 22:36:55.027511 16585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:55.019366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:33765 (local address 127.6.246.193:57477)
0925 22:36:55.020112 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:55.020162 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:55.021657 (+ 1495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:55.022174 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:55.022181 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:55.022263 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:55.022752 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:55.022764 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:55.023568 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:55.023572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:55.024449 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:55.024501 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:55.024999 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:55.025003 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:55.025175 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:55.025879 (+ 704us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:36:55.025881 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:55.025924 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:36:55.025987 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:36:55.026261 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:36:55.026272 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:36:55.026281 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 22:36:55.026421 (+ 140us) client_negotiation.cc:240] Negotiation successful
0925 22:36:55.026693 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":125,"threads_started":1}
I20260925 22:36:55.028885 16583 heartbeater.cc:347] Connected to a master server at 127.6.246.254:33765
I20260925 22:36:55.029114 16583 heartbeater.cc:464] Registering TS with master...
I20260925 22:36:55.029631 16583 heartbeater.cc:511] Master 127.6.246.254:33765 requested a full tablet report, sending...
I20260925 22:36:55.031602 16395 ts_manager.cc:194] Registered new tserver with Master: 56733672a82e42b4855a4624237c68ee (127.6.246.193:46113)
I20260925 22:36:55.033463 16395 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.246.193:57477
W20260925 22:36:55.137315 16586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:36:55.137573 16586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:36:55.137610 16586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:36:55.141713 16586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:36:55.141764 16586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:36:55.141865 16586 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.194
I20260925 22:36:55.146543 16586 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:45181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.246.194
--webserver_port=0
--tserver_master_addrs=127.6.246.254:33765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16586
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.194
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:36:55.147562 16586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:36:55.148965 16586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:36:55.158229 16591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:55.158218 16592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:55.158218 16594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:55.158484 16586 server_base.cc:1034] running on GCE node
I20260925 22:36:55.158936 16586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:36:55.159600 16586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:36:55.160777 16586 hybrid_clock.cc:648] HybridClock initialized: now 1790375815160755 us; error 73 us; skew 500 ppm
I20260925 22:36:55.162729 16586 webserver.cc:460] Webserver started at http://127.6.246.194:33761/ using document root <none> and password file <none>
I20260925 22:36:55.163329 16586 fs_manager.cc:359] Metadata directory not provided
I20260925 22:36:55.163422 16586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:36:55.163623 16586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:36:55.165696 16586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-1/data/instance:
uuid: "89497241e3b34c85bb136002b540cf7c"
format_stamp: "Formatted at 2026-09-25 22:36:55 on dist-test-slave-wjdr"
server_key: "0b41ae638d79e503f1bbf52c4d49d63f"
server_key_iv: "ee01c013712735d69ff737f17b70b35e"
server_key_version: "encryptionkey@0"
I20260925 22:36:55.166265 16586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-1/wal/instance:
uuid: "89497241e3b34c85bb136002b540cf7c"
format_stamp: "Formatted at 2026-09-25 22:36:55 on dist-test-slave-wjdr"
server_key: "0b41ae638d79e503f1bbf52c4d49d63f"
server_key_iv: "ee01c013712735d69ff737f17b70b35e"
server_key_version: "encryptionkey@0"
I20260925 22:36:55.169982 16586 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 22:36:55.172600 16600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:55.174021 16586 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:36:55.174145 16586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-1/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-1/wal
uuid: "89497241e3b34c85bb136002b540cf7c"
format_stamp: "Formatted at 2026-09-25 22:36:55 on dist-test-slave-wjdr"
server_key: "0b41ae638d79e503f1bbf52c4d49d63f"
server_key_iv: "ee01c013712735d69ff737f17b70b35e"
server_key_version: "encryptionkey@0"
I20260925 22:36:55.174258 16586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:36:55.185685 16586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:36:55.186404 16586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:36:55.186601 16586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:36:55.187186 16586 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:36:55.188140 16586 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:36:55.188210 16586 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:55.188279 16586 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:36:55.188311 16586 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:55.197131 16586 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.194:40751
I20260925 22:36:55.197213 16713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.194:40751 every 8 connection(s)
I20260925 22:36:55.198066 16586 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
I20260925 22:36:55.198854 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 16586
I20260925 22:36:55.198971 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-1/wal/instance
I20260925 22:36:55.199504 7131 external_mini_cluster.cc:1428] Setting key 216b8449a753cf29db91df066763fc15
I20260925 22:36:55.201726 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.195:0
--local_ip_for_outbound_sockets=127.6.246.195
--webserver_interface=127.6.246.195
--webserver_port=0
--tserver_master_addrs=127.6.246.254:33765
--builtin_ntp_servers=127.6.246.212:45181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 22:36:55.208643 16434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:55.200205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.194:36263 (local address 127.6.246.254:33765)
0925 22:36:55.200363 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:55.200368 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:55.201579 (+ 1211us) server_negotiation.cc:407] Connection header received
0925 22:36:55.203038 (+ 1459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:55.203041 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:55.203093 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:55.203173 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:55.204023 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:55.204599 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:55.206453 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:55.206662 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:55.206749 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:55.208055 (+ 1306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:55.208058 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:55.208060 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:36:55.208072 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:55.208090 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:36:55.208097 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:55.208209 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:55.208434 (+ 225us) server_negotiation.cc:299] Negotiation successful
0925 22:36:55.208535 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 22:36:55.209829 16716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:55.200748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:33765 (local address 127.6.246.194:36263)
0925 22:36:55.201443 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:55.201484 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:55.202838 (+ 1354us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:55.203330 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:55.203336 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:55.203414 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:55.203881 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:55.203892 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:55.204720 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:55.204726 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:55.206320 (+ 1594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:55.206336 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:55.206826 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:55.206834 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:55.207087 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:55.207735 (+ 648us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:36:55.207738 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:55.207805 (+ 67us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:36:55.207892 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:36:55.208214 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:36:55.208224 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:36:55.208231 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 22:36:55.208381 (+ 150us) client_negotiation.cc:240] Negotiation successful
0925 22:36:55.208719 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":115,"threads_started":1}
I20260925 22:36:55.210973 16714 heartbeater.cc:347] Connected to a master server at 127.6.246.254:33765
I20260925 22:36:55.211264 16714 heartbeater.cc:464] Registering TS with master...
I20260925 22:36:55.211987 16714 heartbeater.cc:511] Master 127.6.246.254:33765 requested a full tablet report, sending...
I20260925 22:36:55.213124 16395 ts_manager.cc:194] Registered new tserver with Master: 89497241e3b34c85bb136002b540cf7c (127.6.246.194:40751)
I20260925 22:36:55.213678 16395 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.246.194:36263
W20260925 22:36:55.314996 16717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:36:55.315276 16717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:36:55.315338 16717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:36:55.319483 16717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:36:55.319566 16717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:36:55.319650 16717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.195
I20260925 22:36:55.324357 16717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:45181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.246.195
--webserver_port=0
--tserver_master_addrs=127.6.246.254:33765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16717
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.195
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:36:55.325457 16717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:36:55.326861 16717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:36:55.336257 16725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:55.336261 16723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:55.336288 16722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:55.336552 16717 server_base.cc:1034] running on GCE node
I20260925 22:36:55.336946 16717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:36:55.337563 16717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:36:55.338716 16717 hybrid_clock.cc:648] HybridClock initialized: now 1790375815338673 us; error 61 us; skew 500 ppm
I20260925 22:36:55.340447 16717 webserver.cc:460] Webserver started at http://127.6.246.195:42695/ using document root <none> and password file <none>
I20260925 22:36:55.341046 16717 fs_manager.cc:359] Metadata directory not provided
I20260925 22:36:55.341109 16717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:36:55.341353 16717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:36:55.343343 16717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-2/data/instance:
uuid: "b5c05c0cb93b4e43a7c895a48e87d96d"
format_stamp: "Formatted at 2026-09-25 22:36:55 on dist-test-slave-wjdr"
server_key: "e28f486fc9b90de71bb1859d61961faf"
server_key_iv: "7f8ff500648a3fbfa0b70a67f6aa0881"
server_key_version: "encryptionkey@0"
I20260925 22:36:55.343888 16717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-2/wal/instance:
uuid: "b5c05c0cb93b4e43a7c895a48e87d96d"
format_stamp: "Formatted at 2026-09-25 22:36:55 on dist-test-slave-wjdr"
server_key: "e28f486fc9b90de71bb1859d61961faf"
server_key_iv: "7f8ff500648a3fbfa0b70a67f6aa0881"
server_key_version: "encryptionkey@0"
I20260925 22:36:55.347404 16717 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 22:36:55.349795 16731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:55.351292 16717 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 22:36:55.351439 16717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-2/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-2/wal
uuid: "b5c05c0cb93b4e43a7c895a48e87d96d"
format_stamp: "Formatted at 2026-09-25 22:36:55 on dist-test-slave-wjdr"
server_key: "e28f486fc9b90de71bb1859d61961faf"
server_key_iv: "7f8ff500648a3fbfa0b70a67f6aa0881"
server_key_version: "encryptionkey@0"
I20260925 22:36:55.351557 16717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:36:55.366080 16717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:36:55.366907 16717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:36:55.367107 16717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:36:55.367812 16717 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:36:55.369114 16717 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:36:55.369179 16717 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:55.369232 16717 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:36:55.369266 16717 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:55.381654 16717 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.195:42939
I20260925 22:36:55.381660 16844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.195:42939 every 8 connection(s)
I20260925 22:36:55.382870 16717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
I20260925 22:36:55.388121 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 16717
I20260925 22:36:55.388247 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-2/wal/instance
I20260925 22:36:55.388566 7131 external_mini_cluster.cc:1428] Setting key c8a56245e39327cd319bafb74bbc3585
I20260925 22:36:55.393157 16434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:55.384818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.195:37499 (local address 127.6.246.254:33765)
0925 22:36:55.384977 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:55.384981 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:55.386251 (+ 1270us) server_negotiation.cc:407] Connection header received
0925 22:36:55.387649 (+ 1398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:55.387652 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:55.387711 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:55.387813 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:55.388840 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:55.389578 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:55.391101 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:55.391350 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:55.391459 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:55.392637 (+ 1178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:55.392640 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:55.392643 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:36:55.392656 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:55.392680 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:36:55.392688 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:55.392759 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:55.392961 (+ 202us) server_negotiation.cc:299] Negotiation successful
0925 22:36:55.393019 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 22:36:55.394035 16847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:55.385372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:33765 (local address 127.6.246.195:37499)
0925 22:36:55.386104 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:55.386144 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:55.387448 (+ 1304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:55.387994 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:55.388001 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:55.388082 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:55.388706 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:55.388721 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:55.389686 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:55.389689 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:55.390982 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:55.390991 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:55.391497 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:55.391501 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:55.391672 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:55.392340 (+ 668us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:36:55.392342 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:55.392381 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:36:55.392441 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:36:55.392773 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:36:55.392784 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:36:55.392790 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 22:36:55.392961 (+ 171us) client_negotiation.cc:240] Negotiation successful
0925 22:36:55.393237 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":149,"threads_started":1}
I20260925 22:36:55.395084 16845 heartbeater.cc:347] Connected to a master server at 127.6.246.254:33765
I20260925 22:36:55.395332 16845 heartbeater.cc:464] Registering TS with master...
I20260925 22:36:55.395946 16845 heartbeater.cc:511] Master 127.6.246.254:33765 requested a full tablet report, sending...
I20260925 22:36:55.397073 16395 ts_manager.cc:194] Registered new tserver with Master: b5c05c0cb93b4e43a7c895a48e87d96d (127.6.246.195:42939)
I20260925 22:36:55.397872 16395 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.246.195:37499
I20260925 22:36:55.406021 7131 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 22:36:55.409132 16434 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:36:55.408113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35396 (local address 127.6.246.254:33765)
0925 22:36:55.408254 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:55.408258 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:55.408429 (+ 171us) server_negotiation.cc:407] Connection header received
0925 22:36:55.408588 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:55.408590 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:55.408629 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:55.408729 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:55.408961 (+ 232us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35396: BlockingRecv error: recv got EOF from 127.0.0.1:35396 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 22:36:55.409534 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 16455
I20260925 22:36:55.415776 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 16586
I20260925 22:36:55.421279 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 16717
I20260925 22:36:55.426438 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 16364
2026-09-25T22:36:55Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (914 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-25T22:36:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:36:55Z Disabled control of system clock
I20260925 22:36:55.455377 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790375752797504-7131-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790375752797504-7131-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790375752797504-7131-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:36879
--webserver_interface=127.6.246.254
--webserver_port=0
--builtin_ntp_servers=127.6.246.212:37929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.246.254:36879
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 22:36:55.563783 16856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:36:55.564054 16856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:36:55.564091 16856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:36:55.568251 16856 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:36:55.568324 16856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:36:55.568339 16856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:36:55.568356 16856 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:36:55.568377 16856 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:36:55.573345 16856 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:37929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790375752797504-7131-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790375752797504-7131-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.246.254:36879
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:36879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.246.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790375752797504-7131-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:36:55.574519 16856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:36:55.575917 16856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:36:55.584329 16861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:55.584352 16864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:55.584331 16862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:55.584554 16856 server_base.cc:1034] running on GCE node
I20260925 22:36:55.585179 16856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:36:55.586246 16856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:36:55.587428 16856 hybrid_clock.cc:648] HybridClock initialized: now 1790375815587405 us; error 43 us; skew 500 ppm
I20260925 22:36:55.589275 16856 webserver.cc:460] Webserver started at http://127.6.246.254:35315/ using document root <none> and password file <none>
I20260925 22:36:55.589818 16856 fs_manager.cc:359] Metadata directory not provided
I20260925 22:36:55.589902 16856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:36:55.590142 16856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:36:55.592083 16856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790375752797504-7131-0/minicluster-data/master-0/data/instance:
uuid: "2527098b58d34cecba2baec87374c36f"
format_stamp: "Formatted at 2026-09-25 22:36:55 on dist-test-slave-wjdr"
server_key: "d7ba89f4644dcc4b57ae06c354cb89d3"
server_key_iv: "046fedf0cae20f2fbc83b0c1e5edefae"
server_key_version: "encryptionkey@0"
I20260925 22:36:55.592667 16856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790375752797504-7131-0/minicluster-data/master-0/wal/instance:
uuid: "2527098b58d34cecba2baec87374c36f"
format_stamp: "Formatted at 2026-09-25 22:36:55 on dist-test-slave-wjdr"
server_key: "d7ba89f4644dcc4b57ae06c354cb89d3"
server_key_iv: "046fedf0cae20f2fbc83b0c1e5edefae"
server_key_version: "encryptionkey@0"
I20260925 22:36:55.596170 16856 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:36:55.598459 16870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:55.599586 16856 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:36:55.599714 16856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790375752797504-7131-0/minicluster-data/master-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790375752797504-7131-0/minicluster-data/master-0/wal
uuid: "2527098b58d34cecba2baec87374c36f"
format_stamp: "Formatted at 2026-09-25 22:36:55 on dist-test-slave-wjdr"
server_key: "d7ba89f4644dcc4b57ae06c354cb89d3"
server_key_iv: "046fedf0cae20f2fbc83b0c1e5edefae"
server_key_version: "encryptionkey@0"
I20260925 22:36:55.599867 16856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790375752797504-7131-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790375752797504-7131-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790375752797504-7131-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:36:55.604211 16856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:36:55.604832 16856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:36:55.605003 16856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:36:55.611410 16856 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.254:36879
I20260925 22:36:55.611434 16922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.254:36879 every 8 connection(s)
I20260925 22:36:55.612285 16856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
I20260925 22:36:55.616034 16923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:36:55.620726 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 16856
I20260925 22:36:55.620882 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790375752797504-7131-0/minicluster-data/master-0/wal/instance
I20260925 22:36:55.621150 7131 external_mini_cluster.cc:1428] Setting key fd90a3de4e67e6617d842ce97ee1a3f9
I20260925 22:36:55.621815 16923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2527098b58d34cecba2baec87374c36f: Bootstrap starting.
I20260925 22:36:55.624271 16923 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2527098b58d34cecba2baec87374c36f: Neither blocks nor log segments found. Creating new log.
I20260925 22:36:55.625195 16923 log.cc:824] T 00000000000000000000000000000000 P 2527098b58d34cecba2baec87374c36f: Log is configured to *not* fsync() on all Append() calls
I20260925 22:36:55.626530 16926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:55.622496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51336 (local address 127.6.246.254:36879)
0925 22:36:55.623135 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:55.623151 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:55.623186 (+ 35us) server_negotiation.cc:407] Connection header received
0925 22:36:55.623814 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:55.623829 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:55.624166 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:55.624522 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:55.625327 (+ 805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:55.625335 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:55.625350 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:36:55.625364 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:55.625428 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:36:55.625446 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:55.625557 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:55.625654 (+ 97us) server_negotiation.cc:299] Negotiation successful
0925 22:36:55.625885 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":125,"threads_started":1}
I20260925 22:36:55.627156 16923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2527098b58d34cecba2baec87374c36f: No bootstrap required, opened a new log
I20260925 22:36:55.630753 16923 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2527098b58d34cecba2baec87374c36f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2527098b58d34cecba2baec87374c36f" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 36879 } }
I20260925 22:36:55.631006 16923 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2527098b58d34cecba2baec87374c36f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:36:55.631078 16923 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2527098b58d34cecba2baec87374c36f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2527098b58d34cecba2baec87374c36f, State: Initialized, Role: FOLLOWER
I20260925 22:36:55.631507 16923 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2527098b58d34cecba2baec87374c36f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2527098b58d34cecba2baec87374c36f" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 36879 } }
I20260925 22:36:55.631657 16923 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2527098b58d34cecba2baec87374c36f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:36:55.631771 16923 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2527098b58d34cecba2baec87374c36f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:36:55.631901 16923 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2527098b58d34cecba2baec87374c36f [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:36:55.632987 16923 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2527098b58d34cecba2baec87374c36f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2527098b58d34cecba2baec87374c36f" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 36879 } }
I20260925 22:36:55.633353 16923 leader_election.cc:302] T 00000000000000000000000000000000 P 2527098b58d34cecba2baec87374c36f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2527098b58d34cecba2baec87374c36f; no voters:
I20260925 22:36:55.633682 16923 leader_election.cc:288] T 00000000000000000000000000000000 P 2527098b58d34cecba2baec87374c36f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:36:55.633863 16928 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2527098b58d34cecba2baec87374c36f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:36:55.634119 16928 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2527098b58d34cecba2baec87374c36f [term 1 LEADER]: Becoming Leader. State: Replica: 2527098b58d34cecba2baec87374c36f, State: Running, Role: LEADER
I20260925 22:36:55.634474 16928 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2527098b58d34cecba2baec87374c36f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2527098b58d34cecba2baec87374c36f" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 36879 } }
I20260925 22:36:55.635198 16923 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2527098b58d34cecba2baec87374c36f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:36:55.636186 16929 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2527098b58d34cecba2baec87374c36f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2527098b58d34cecba2baec87374c36f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2527098b58d34cecba2baec87374c36f" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 36879 } } }
I20260925 22:36:55.636315 16929 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2527098b58d34cecba2baec87374c36f [sys.catalog]: This master's current role is: LEADER
I20260925 22:36:55.636343 16930 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2527098b58d34cecba2baec87374c36f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2527098b58d34cecba2baec87374c36f. Latest consensus state: current_term: 1 leader_uuid: "2527098b58d34cecba2baec87374c36f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2527098b58d34cecba2baec87374c36f" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 36879 } } }
I20260925 22:36:55.636492 16930 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2527098b58d34cecba2baec87374c36f [sys.catalog]: This master's current role is: LEADER
I20260925 22:36:55.636858 16937 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:36:55.640043 16937 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:36:55.645967 16937 catalog_manager.cc:1347] Generated new cluster ID: 27bde04c54184733a1b92f9f1d0b1681
I20260925 22:36:55.646036 16937 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:36:55.660780 16937 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:36:55.661150 16937 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:36:55.669879 16937 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2527098b58d34cecba2baec87374c36f: Generated new TSK 0
I20260925 22:36:55.670604 16937 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:36:55.678953 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.193:0
--local_ip_for_outbound_sockets=127.6.246.193
--webserver_interface=127.6.246.193
--webserver_port=0
--tserver_master_addrs=127.6.246.254:36879
--builtin_ntp_servers=127.6.246.212:37929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 22:36:55.797049 16947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:36:55.797273 16947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:36:55.797309 16947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:36:55.801331 16947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:36:55.801376 16947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:36:55.801457 16947 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.193
I20260925 22:36:55.806104 16947 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:37929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.246.193
--webserver_port=0
--tserver_master_addrs=127.6.246.254:36879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16947
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.193
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:36:55.807168 16947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:36:55.808538 16947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:36:55.817821 16953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:55.817824 16955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:55.817832 16952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:55.818007 16947 server_base.cc:1034] running on GCE node
I20260925 22:36:55.818601 16947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:36:55.819206 16947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:36:55.820389 16947 hybrid_clock.cc:648] HybridClock initialized: now 1790375815820362 us; error 44 us; skew 500 ppm
I20260925 22:36:55.822214 16947 webserver.cc:460] Webserver started at http://127.6.246.193:45601/ using document root <none> and password file <none>
I20260925 22:36:55.822774 16947 fs_manager.cc:359] Metadata directory not provided
I20260925 22:36:55.822867 16947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:36:55.823081 16947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:36:55.825165 16947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-0/data/instance:
uuid: "b033f332484e4bcbbaab3fc8e5d648ba"
format_stamp: "Formatted at 2026-09-25 22:36:55 on dist-test-slave-wjdr"
server_key: "b52b79fa732c6dfbd0f4fb1f446294dd"
server_key_iv: "f68148913cb4f66fc3cd3426afd41658"
server_key_version: "encryptionkey@0"
I20260925 22:36:55.825716 16947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-0/wal/instance:
uuid: "b033f332484e4bcbbaab3fc8e5d648ba"
format_stamp: "Formatted at 2026-09-25 22:36:55 on dist-test-slave-wjdr"
server_key: "b52b79fa732c6dfbd0f4fb1f446294dd"
server_key_iv: "f68148913cb4f66fc3cd3426afd41658"
server_key_version: "encryptionkey@0"
I20260925 22:36:55.829195 16947 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:36:55.831529 16961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:55.832753 16947 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260925 22:36:55.832877 16947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-0/wal
uuid: "b033f332484e4bcbbaab3fc8e5d648ba"
format_stamp: "Formatted at 2026-09-25 22:36:55 on dist-test-slave-wjdr"
server_key: "b52b79fa732c6dfbd0f4fb1f446294dd"
server_key_iv: "f68148913cb4f66fc3cd3426afd41658"
server_key_version: "encryptionkey@0"
I20260925 22:36:55.832990 16947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:36:55.837724 16947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:36:55.838290 16947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:36:55.838486 16947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:36:55.839003 16947 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:36:55.839851 16947 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:36:55.839916 16947 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:55.839978 16947 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:36:55.840024 16947 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:55.848572 16947 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.193:45699
I20260925 22:36:55.848610 17074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.193:45699 every 8 connection(s)
I20260925 22:36:55.849428 16947 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
I20260925 22:36:55.855196 16926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:55.851221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.193:51667 (local address 127.6.246.254:36879)
0925 22:36:55.851392 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:55.851396 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:55.852347 (+ 951us) server_negotiation.cc:407] Connection header received
0925 22:36:55.853675 (+ 1328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:55.853677 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:55.853713 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:55.853823 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:55.854805 (+ 982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:55.854808 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:55.854810 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:36:55.854821 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:55.854840 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:36:55.854847 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:55.854948 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:55.855059 (+ 111us) server_negotiation.cc:299] Negotiation successful
0925 22:36:55.855093 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 22:36:55.855798 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 16947
I20260925 22:36:55.855923 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-0/wal/instance
I20260925 22:36:55.856259 7131 external_mini_cluster.cc:1428] Setting key 9f0153d0590647d1faded1356e48bef7
I20260925 22:36:55.856259 17077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:55.851582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:36879 (local address 127.6.246.193:51667)
0925 22:36:55.852217 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:55.852255 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:55.853490 (+ 1235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:55.853960 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:55.853966 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:55.854080 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:55.854576 (+ 496us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:36:55.854577 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:55.854615 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:36:55.854678 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:36:55.854942 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:36:55.854948 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:36:55.854952 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 22:36:55.855077 (+ 125us) client_negotiation.cc:240] Negotiation successful
0925 22:36:55.855276 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":103,"threads_started":1}
I20260925 22:36:55.857434 17075 heartbeater.cc:347] Connected to a master server at 127.6.246.254:36879
I20260925 22:36:55.857658 17075 heartbeater.cc:464] Registering TS with master...
I20260925 22:36:55.858091 17075 heartbeater.cc:511] Master 127.6.246.254:36879 requested a full tablet report, sending...
I20260925 22:36:55.858430 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.194:0
--local_ip_for_outbound_sockets=127.6.246.194
--webserver_interface=127.6.246.194
--webserver_port=0
--tserver_master_addrs=127.6.246.254:36879
--builtin_ntp_servers=127.6.246.212:37929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260925 22:36:55.859613 16886 ts_manager.cc:194] Registered new tserver with Master: b033f332484e4bcbbaab3fc8e5d648ba (127.6.246.193:45699)
W20260925 22:36:55.969506 17078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:36:55.969769 17078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:36:55.969816 17078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:36:55.973891 17078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:36:55.973944 17078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:36:55.974000 17078 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.194
I20260925 22:36:55.978626 17078 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:37929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.246.194
--webserver_port=0
--tserver_master_addrs=127.6.246.254:36879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17078
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.194
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:36:55.979615 17078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:36:55.980942 17078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:36:55.989988 17083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:55.989982 17084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:55.989984 17086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:55.990286 17078 server_base.cc:1034] running on GCE node
I20260925 22:36:55.990664 17078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:36:55.991279 17078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:36:55.992446 17078 hybrid_clock.cc:648] HybridClock initialized: now 1790375815992415 us; error 43 us; skew 500 ppm
I20260925 22:36:55.994239 17078 webserver.cc:460] Webserver started at http://127.6.246.194:34935/ using document root <none> and password file <none>
I20260925 22:36:55.994793 17078 fs_manager.cc:359] Metadata directory not provided
I20260925 22:36:55.994876 17078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:36:55.995093 17078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:36:55.997195 17078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-1/data/instance:
uuid: "c0a49247358748f990e5572ce3f6882e"
format_stamp: "Formatted at 2026-09-25 22:36:55 on dist-test-slave-wjdr"
server_key: "b1646e68817bddfbe1da0935cc9b8454"
server_key_iv: "818b4d485f9a4c409a395ea8a940b1ab"
server_key_version: "encryptionkey@0"
I20260925 22:36:55.997763 17078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-1/wal/instance:
uuid: "c0a49247358748f990e5572ce3f6882e"
format_stamp: "Formatted at 2026-09-25 22:36:55 on dist-test-slave-wjdr"
server_key: "b1646e68817bddfbe1da0935cc9b8454"
server_key_iv: "818b4d485f9a4c409a395ea8a940b1ab"
server_key_version: "encryptionkey@0"
I20260925 22:36:56.001397 17078 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:36:56.003912 17092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:56.005331 17078 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 22:36:56.005478 17078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-1/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-1/wal
uuid: "c0a49247358748f990e5572ce3f6882e"
format_stamp: "Formatted at 2026-09-25 22:36:55 on dist-test-slave-wjdr"
server_key: "b1646e68817bddfbe1da0935cc9b8454"
server_key_iv: "818b4d485f9a4c409a395ea8a940b1ab"
server_key_version: "encryptionkey@0"
I20260925 22:36:56.005597 17078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:36:56.010169 17078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:36:56.010785 17078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:36:56.010965 17078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:36:56.011497 17078 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:36:56.012375 17078 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:36:56.012441 17078 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:56.012548 17078 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:36:56.012596 17078 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:56.021552 17078 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.194:45993
I20260925 22:36:56.021590 17205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.194:45993 every 8 connection(s)
I20260925 22:36:56.022436 17078 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
I20260925 22:36:56.024850 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 17078
I20260925 22:36:56.025030 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-1/wal/instance
I20260925 22:36:56.025363 7131 external_mini_cluster.cc:1428] Setting key 9b4e4442ab51f7d1cbf0231fe6b1ae7e
I20260925 22:36:56.027509 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.195:0
--local_ip_for_outbound_sockets=127.6.246.195
--webserver_interface=127.6.246.195
--webserver_port=0
--tserver_master_addrs=127.6.246.254:36879
--builtin_ntp_servers=127.6.246.212:37929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260925 22:36:56.029136 16926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:56.024276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.194:51217 (local address 127.6.246.254:36879)
0925 22:36:56.024433 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:56.024437 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:56.025761 (+ 1324us) server_negotiation.cc:407] Connection header received
0925 22:36:56.027328 (+ 1567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:56.027330 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:56.027363 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:56.027424 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:56.028622 (+ 1198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:56.028625 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:56.028627 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:36:56.028637 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:56.028657 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:36:56.028664 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:56.028734 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:56.028914 (+ 180us) server_negotiation.cc:299] Negotiation successful
0925 22:36:56.028963 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 22:36:56.029963 17208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:56.024729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:36879 (local address 127.6.246.194:51217)
0925 22:36:56.025571 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:56.025627 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:56.027101 (+ 1474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:56.027621 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:56.027631 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:56.027744 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:56.028315 (+ 571us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:36:56.028319 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:56.028381 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:36:56.028490 (+ 109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:36:56.028746 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:36:56.028756 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:36:56.028764 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 22:36:56.028940 (+ 176us) client_negotiation.cc:240] Negotiation successful
0925 22:36:56.029204 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":127,"threads_started":1}
I20260925 22:36:56.031077 17206 heartbeater.cc:347] Connected to a master server at 127.6.246.254:36879
I20260925 22:36:56.031311 17206 heartbeater.cc:464] Registering TS with master...
I20260925 22:36:56.031790 17206 heartbeater.cc:511] Master 127.6.246.254:36879 requested a full tablet report, sending...
I20260925 22:36:56.032995 16886 ts_manager.cc:194] Registered new tserver with Master: c0a49247358748f990e5572ce3f6882e (127.6.246.194:45993)
W20260925 22:36:56.140403 17209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:36:56.140717 17209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:36:56.140774 17209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:36:56.144771 17209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:36:56.144819 17209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:36:56.144905 17209 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.195
I20260925 22:36:56.149559 17209 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:37929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.246.195
--webserver_port=0
--tserver_master_addrs=127.6.246.254:36879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.195
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:36:56.150588 17209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:36:56.151903 17209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:36:56.161700 17217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:56.161885 17209 server_base.cc:1034] running on GCE node
W20260925 22:36:56.161698 17215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:56.161692 17214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:56.162384 17209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:36:56.163029 17209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:36:56.164204 17209 hybrid_clock.cc:648] HybridClock initialized: now 1790375816164190 us; error 26 us; skew 500 ppm
I20260925 22:36:56.166024 17209 webserver.cc:460] Webserver started at http://127.6.246.195:36829/ using document root <none> and password file <none>
I20260925 22:36:56.166586 17209 fs_manager.cc:359] Metadata directory not provided
I20260925 22:36:56.166671 17209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:36:56.166878 17209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:36:56.168937 17209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-2/data/instance:
uuid: "d2845821b9d44f3d9af31873dd48f599"
format_stamp: "Formatted at 2026-09-25 22:36:56 on dist-test-slave-wjdr"
server_key: "57a8bc65cee7acc055fb1111e2e4a99d"
server_key_iv: "0ea00fb1cc5440c12b68ffcd177844ad"
server_key_version: "encryptionkey@0"
I20260925 22:36:56.169480 17209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-2/wal/instance:
uuid: "d2845821b9d44f3d9af31873dd48f599"
format_stamp: "Formatted at 2026-09-25 22:36:56 on dist-test-slave-wjdr"
server_key: "57a8bc65cee7acc055fb1111e2e4a99d"
server_key_iv: "0ea00fb1cc5440c12b68ffcd177844ad"
server_key_version: "encryptionkey@0"
I20260925 22:36:56.172998 17209 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:36:56.175483 17223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:56.176676 17209 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:36:56.176810 17209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-2/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-2/wal
uuid: "d2845821b9d44f3d9af31873dd48f599"
format_stamp: "Formatted at 2026-09-25 22:36:56 on dist-test-slave-wjdr"
server_key: "57a8bc65cee7acc055fb1111e2e4a99d"
server_key_iv: "0ea00fb1cc5440c12b68ffcd177844ad"
server_key_version: "encryptionkey@0"
I20260925 22:36:56.176944 17209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:36:56.181407 17209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:36:56.181985 17209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:36:56.182153 17209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:36:56.182653 17209 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:36:56.183499 17209 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:36:56.183563 17209 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:56.183624 17209 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:36:56.183670 17209 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:56.192382 17209 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.195:35121
I20260925 22:36:56.192410 17336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.195:35121 every 8 connection(s)
I20260925 22:36:56.193254 17209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
I20260925 22:36:56.194298 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 17209
I20260925 22:36:56.194411 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790375752797504-7131-0/minicluster-data/ts-2/wal/instance
I20260925 22:36:56.194695 7131 external_mini_cluster.cc:1428] Setting key 7d82964fe4cd86ea7fd13b3bc8ce83b7
I20260925 22:36:56.199308 16926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:56.194982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.195:42815 (local address 127.6.246.254:36879)
0925 22:36:56.195102 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:56.195110 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:56.196379 (+ 1269us) server_negotiation.cc:407] Connection header received
0925 22:36:56.197739 (+ 1360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:56.197742 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:56.197779 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:56.197851 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:56.198940 (+ 1089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:56.198943 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:56.198945 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:36:56.198955 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:56.198974 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:36:56.198982 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:56.199053 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:56.199170 (+ 117us) server_negotiation.cc:299] Negotiation successful
0925 22:36:56.199206 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 22:36:56.200217 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:56.195428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:36879 (local address 127.6.246.195:42815)
0925 22:36:56.196217 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:56.196273 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:56.197557 (+ 1284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:56.198036 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:56.198043 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:56.198129 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:56.198679 (+ 550us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:36:56.198681 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:56.198720 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:36:56.198785 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:36:56.199056 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:36:56.199062 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:36:56.199066 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 22:36:56.199197 (+ 131us) client_negotiation.cc:240] Negotiation successful
0925 22:36:56.199423 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":131,"threads_started":1}
I20260925 22:36:56.201310 17337 heartbeater.cc:347] Connected to a master server at 127.6.246.254:36879
I20260925 22:36:56.201529 17337 heartbeater.cc:464] Registering TS with master...
I20260925 22:36:56.201889 17337 heartbeater.cc:511] Master 127.6.246.254:36879 requested a full tablet report, sending...
I20260925 22:36:56.202811 16886 ts_manager.cc:194] Registered new tserver with Master: d2845821b9d44f3d9af31873dd48f599 (127.6.246.195:35121)
I20260925 22:36:56.208228 7131 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 22:36:56.211026 16926 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:36:56.210156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51350 (local address 127.6.246.254:36879)
0925 22:36:56.210278 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:56.210281 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:56.210482 (+ 201us) server_negotiation.cc:407] Connection header received
0925 22:36:56.210552 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:56.210554 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:56.210586 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:56.210656 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:56.210901 (+ 245us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51350: BlockingRecv error: recv got EOF from 127.0.0.1:51350 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 22:36:56.211292 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 16947
I20260925 22:36:56.217106 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 17078
I20260925 22:36:56.222275 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 17209
I20260925 22:36:56.227273 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 16856
2026-09-25T22:36:56Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (800 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:36:56 dist-test-slave-wjdr krb5kdc[17347](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:36:56 dist-test-slave-wjdr krb5kdc[17347](info): set up 2 sockets
Sep 25 22:36:56 dist-test-slave-wjdr krb5kdc[17347](info): commencing operation
krb5kdc: starting...
W20260925 22:36:57.273910 7131 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Sep 25 22:36:57 dist-test-slave-wjdr krb5kdc[17347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:36:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:36:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.246.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab.
Entry for principal kudu/127.6.246.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab.
I20260925 22:36:57.401649 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:35903
--webserver_interface=127.6.246.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab
--principal=kudu/127.6.246.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:33253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.246.254:35903
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:36:57.526912 17360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:36:57.527226 17360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:36:57.527267 17360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:36:57.531369 17360 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:36:57.531428 17360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:36:57.531450 17360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:36:57.531466 17360 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:36:57.531481 17360 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:36:57.537774 17360 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:33253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.246.254:35903
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab
--principal=kudu/127.6.246.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:35903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.246.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17360
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:36:57.539215 17360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:36:57.541025 17360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:36:57.554852 17368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:57.554924 17366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:57.554855 17365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:57.555192 17360 server_base.cc:1034] running on GCE node
I20260925 22:36:57.556114 17360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:36:57.558023 17360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:36:57.559209 17360 hybrid_clock.cc:648] HybridClock initialized: now 1790375817559190 us; error 33 us; skew 500 ppm
Sep 25 22:36:57 dist-test-slave-wjdr krb5kdc[17347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375817, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:36:57.562928 17360 init.cc:377] Logged in from keytab as kudu/127.6.246.254@KRBTEST.COM (short username kudu)
I20260925 22:36:57.564247 17360 webserver.cc:460] Webserver started at http://127.6.246.254:45189/ using document root <none> and password file <none>
I20260925 22:36:57.565145 17360 fs_manager.cc:359] Metadata directory not provided
I20260925 22:36:57.565250 17360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:36:57.565578 17360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:36:57.569051 17360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/minicluster-data/master-0/data/instance:
uuid: "02b783ccb91a4c4b8f5e9ad898cd8582"
format_stamp: "Formatted at 2026-09-25 22:36:57 on dist-test-slave-wjdr"
server_key: "a7bdcdb14d33f6d66449cbab9c493f26"
server_key_iv: "520e30c32e1349622394f8514f0b148e"
server_key_version: "encryptionkey@0"
I20260925 22:36:57.569808 17360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/minicluster-data/master-0/wal/instance:
uuid: "02b783ccb91a4c4b8f5e9ad898cd8582"
format_stamp: "Formatted at 2026-09-25 22:36:57 on dist-test-slave-wjdr"
server_key: "a7bdcdb14d33f6d66449cbab9c493f26"
server_key_iv: "520e30c32e1349622394f8514f0b148e"
server_key_version: "encryptionkey@0"
I20260925 22:36:57.575295 17360 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 22:36:57.578493 17375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:57.580341 17360 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:36:57.580528 17360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/minicluster-data/master-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/minicluster-data/master-0/wal
uuid: "02b783ccb91a4c4b8f5e9ad898cd8582"
format_stamp: "Formatted at 2026-09-25 22:36:57 on dist-test-slave-wjdr"
server_key: "a7bdcdb14d33f6d66449cbab9c493f26"
server_key_iv: "520e30c32e1349622394f8514f0b148e"
server_key_version: "encryptionkey@0"
I20260925 22:36:57.580677 17360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:36:57.600708 17360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:36:57.609692 17360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:36:57.609923 17360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:36:57.618360 17360 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.254:35903
I20260925 22:36:57.618695 17427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.254:35903 every 8 connection(s)
I20260925 22:36:57.619583 17360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
I20260925 22:36:57.623051 17428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:36:57.629400 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 17360
I20260925 22:36:57.629513 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/minicluster-data/master-0/wal/instance
I20260925 22:36:57.629524 17428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02b783ccb91a4c4b8f5e9ad898cd8582: Bootstrap starting.
I20260925 22:36:57.629858 7131 external_mini_cluster.cc:1428] Setting key 8d97e79b6719dcfc4e63e181b663150c
I20260925 22:36:57.632556 17428 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02b783ccb91a4c4b8f5e9ad898cd8582: Neither blocks nor log segments found. Creating new log.
I20260925 22:36:57.633421 17428 log.cc:824] T 00000000000000000000000000000000 P 02b783ccb91a4c4b8f5e9ad898cd8582: Log is configured to *not* fsync() on all Append() calls
I20260925 22:36:57.635701 17428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02b783ccb91a4c4b8f5e9ad898cd8582: No bootstrap required, opened a new log
Sep 25 22:36:57 dist-test-slave-wjdr krb5kdc[17347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
I20260925 22:36:57.640719 17428 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02b783ccb91a4c4b8f5e9ad898cd8582 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02b783ccb91a4c4b8f5e9ad898cd8582" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 35903 } }
I20260925 22:36:57.640971 17428 raft_consensus.cc:374] T 00000000000000000000000000000000 P 02b783ccb91a4c4b8f5e9ad898cd8582 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:36:57.641077 17428 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02b783ccb91a4c4b8f5e9ad898cd8582 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02b783ccb91a4c4b8f5e9ad898cd8582, State: Initialized, Role: FOLLOWER
I20260925 22:36:57.641642 17428 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02b783ccb91a4c4b8f5e9ad898cd8582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02b783ccb91a4c4b8f5e9ad898cd8582" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 35903 } }
I20260925 22:36:57.641808 17428 raft_consensus.cc:388] T 00000000000000000000000000000000 P 02b783ccb91a4c4b8f5e9ad898cd8582 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:36:57.641881 17428 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02b783ccb91a4c4b8f5e9ad898cd8582 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:36:57.642004 17428 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02b783ccb91a4c4b8f5e9ad898cd8582 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:36:57.643371 17428 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02b783ccb91a4c4b8f5e9ad898cd8582 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02b783ccb91a4c4b8f5e9ad898cd8582" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 35903 } }
I20260925 22:36:57.643797 17428 leader_election.cc:302] T 00000000000000000000000000000000 P 02b783ccb91a4c4b8f5e9ad898cd8582 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 02b783ccb91a4c4b8f5e9ad898cd8582; no voters:
I20260925 22:36:57.644281 17428 leader_election.cc:288] T 00000000000000000000000000000000 P 02b783ccb91a4c4b8f5e9ad898cd8582 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:36:57.644402 17433 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02b783ccb91a4c4b8f5e9ad898cd8582 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:36:57.644671 17433 raft_consensus.cc:686] T 00000000000000000000000000000000 P 02b783ccb91a4c4b8f5e9ad898cd8582 [term 1 LEADER]: Becoming Leader. State: Replica: 02b783ccb91a4c4b8f5e9ad898cd8582, State: Running, Role: LEADER
I20260925 22:36:57.645027 17433 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02b783ccb91a4c4b8f5e9ad898cd8582 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02b783ccb91a4c4b8f5e9ad898cd8582" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 35903 } }
I20260925 22:36:57.645838 17428 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02b783ccb91a4c4b8f5e9ad898cd8582 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:36:57.646965 17435 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02b783ccb91a4c4b8f5e9ad898cd8582 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02b783ccb91a4c4b8f5e9ad898cd8582. Latest consensus state: current_term: 1 leader_uuid: "02b783ccb91a4c4b8f5e9ad898cd8582" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02b783ccb91a4c4b8f5e9ad898cd8582" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 35903 } } }
I20260925 22:36:57.647089 17435 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02b783ccb91a4c4b8f5e9ad898cd8582 [sys.catalog]: This master's current role is: LEADER
I20260925 22:36:57.647393 17431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:57.631470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59882 (local address 127.6.246.254:35903)
0925 22:36:57.631915 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:57.631932 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:57.631972 (+ 40us) server_negotiation.cc:407] Connection header received
0925 22:36:57.632754 (+ 782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:57.632783 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:57.633152 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:57.633484 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:57.634601 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:57.635703 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:57.636986 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:57.637486 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:57.637613 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:57.640868 (+ 3255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:57.640874 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:57.640889 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:57.640918 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:57.643278 (+ 2360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:57.643773 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:57.643781 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:57.643787 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:57.643975 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:57.644249 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:57.644253 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:57.644255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:57.644874 (+ 619us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:57.645091 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:57.645839 (+ 748us) server_negotiation.cc:299] Negotiation successful
0925 22:36:57.646164 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":147,"threads_started":1}
I20260925 22:36:57.647413 17434 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02b783ccb91a4c4b8f5e9ad898cd8582 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "02b783ccb91a4c4b8f5e9ad898cd8582" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02b783ccb91a4c4b8f5e9ad898cd8582" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 35903 } } }
I20260925 22:36:57.647504 17434 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02b783ccb91a4c4b8f5e9ad898cd8582 [sys.catalog]: This master's current role is: LEADER
I20260925 22:36:57.647761 17442 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:36:57.651217 17442 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:36:57.657896 17442 catalog_manager.cc:1347] Generated new cluster ID: 911f8303cd9f4bc498d0ae46f6076c02
I20260925 22:36:57.657963 17442 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:36:57.662648 17442 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:36:57.663960 17442 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:36:57.670547 17442 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 02b783ccb91a4c4b8f5e9ad898cd8582: Generated new TSK 0
I20260925 22:36:57.671353 17442 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.246.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.193.keytab.
Entry for principal kudu/127.6.246.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.193.keytab.
I20260925 22:36:57.701810 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.193:0
--local_ip_for_outbound_sockets=127.6.246.193
--webserver_interface=127.6.246.193
--webserver_port=0
--tserver_master_addrs=127.6.246.254:35903
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.193.keytab
--principal=kudu/127.6.246.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:33253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:36:57.831319 17454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:36:57.831632 17454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:36:57.831678 17454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:36:57.835872 17454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:36:57.835942 17454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:36:57.836061 17454 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.193
I20260925 22:36:57.841704 17454 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:33253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.193.keytab
--principal=kudu/127.6.246.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.246.193
--webserver_port=0
--tserver_master_addrs=127.6.246.254:35903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.193
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:36:57.843026 17454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:36:57.844308 17454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:36:57.855769 17460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:57.855837 17459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:57.855769 17462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:57.856179 17454 server_base.cc:1034] running on GCE node
I20260925 22:36:57.856665 17454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:36:57.857342 17454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:36:57.858495 17454 hybrid_clock.cc:648] HybridClock initialized: now 1790375817858466 us; error 44 us; skew 500 ppm
Sep 25 22:36:57 dist-test-slave-wjdr krb5kdc[17347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375817, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:36:57.861575 17454 init.cc:377] Logged in from keytab as kudu/127.6.246.193@KRBTEST.COM (short username kudu)
I20260925 22:36:57.862531 17454 webserver.cc:460] Webserver started at http://127.6.246.193:41591/ using document root <none> and password file <none>
I20260925 22:36:57.863188 17454 fs_manager.cc:359] Metadata directory not provided
I20260925 22:36:57.863250 17454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:36:57.863502 17454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:36:57.865912 17454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/minicluster-data/ts-0/data/instance:
uuid: "de318803889c4ccd8bc4ef42bbcd8bce"
format_stamp: "Formatted at 2026-09-25 22:36:57 on dist-test-slave-wjdr"
server_key: "55e09c6fb4396b4b5bde206407fdcc40"
server_key_iv: "feb88504cb2731332106206f63515471"
server_key_version: "encryptionkey@0"
I20260925 22:36:57.866529 17454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/minicluster-data/ts-0/wal/instance:
uuid: "de318803889c4ccd8bc4ef42bbcd8bce"
format_stamp: "Formatted at 2026-09-25 22:36:57 on dist-test-slave-wjdr"
server_key: "55e09c6fb4396b4b5bde206407fdcc40"
server_key_iv: "feb88504cb2731332106206f63515471"
server_key_version: "encryptionkey@0"
I20260925 22:36:57.871325 17454 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:36:57.874851 17469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:57.876276 17454 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:36:57.876411 17454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/minicluster-data/ts-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/minicluster-data/ts-0/wal
uuid: "de318803889c4ccd8bc4ef42bbcd8bce"
format_stamp: "Formatted at 2026-09-25 22:36:57 on dist-test-slave-wjdr"
server_key: "55e09c6fb4396b4b5bde206407fdcc40"
server_key_iv: "feb88504cb2731332106206f63515471"
server_key_version: "encryptionkey@0"
I20260925 22:36:57.876571 17454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:36:57.890528 17454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:36:57.893867 17454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:36:57.894080 17454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:36:57.894753 17454 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:36:57.895705 17454 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:36:57.895773 17454 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:57.895861 17454 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:36:57.895906 17454 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:57.906384 17454 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.193:42073
I20260925 22:36:57.906405 17582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.193:42073 every 8 connection(s)
I20260925 22:36:57.907433 17454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
I20260925 22:36:57.908948 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 17454
I20260925 22:36:57.909077 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/minicluster-data/ts-0/wal/instance
I20260925 22:36:57.909475 7131 external_mini_cluster.cc:1428] Setting key 7fcab6459e13416171f40a4e2dd7e66a
Sep 25 22:36:57 dist-test-slave-wjdr krb5kdc[17347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375817, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.193@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.246.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.194@KRBTEST.COM" created.
I20260925 22:36:57.923223 17431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:57.909611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.193:42403 (local address 127.6.246.254:35903)
0925 22:36:57.909769 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:57.909774 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:57.910863 (+ 1089us) server_negotiation.cc:407] Connection header received
0925 22:36:57.912478 (+ 1615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:57.912484 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:57.912558 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:57.912660 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:57.914035 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:57.914857 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:57.916050 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:57.916395 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:57.916571 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:57.919283 (+ 2712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:57.919287 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:57.919290 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:57.919321 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:57.920889 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:57.921404 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:57.921409 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:57.921452 (+ 43us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:57.921532 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:57.921876 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:57.921881 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:57.921883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:57.922082 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:57.922209 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:57.922812 (+ 603us) server_negotiation.cc:299] Negotiation successful
0925 22:36:57.922960 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 22:36:57.924317 17585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:57.910127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:35903 (local address 127.6.246.193:42403)
0925 22:36:57.910676 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:57.910735 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:57.912191 (+ 1456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:57.912848 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:57.912855 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:57.913254 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:57.913866 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:57.913881 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:57.914973 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:57.914976 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:57.915910 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:57.915919 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:57.916611 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:57.916615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:57.916812 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:57.917519 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:57.917536 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:57.919111 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:36:57.921002 (+ 1891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:57.921008 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:57.921025 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:57.921301 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:57.921630 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:57.921633 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:57.921635 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:57.921783 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:57.922216 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:36:57.922224 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:36:57.922523 (+ 299us) client_negotiation.cc:769] Sending connection context
0925 22:36:57.922766 (+ 243us) client_negotiation.cc:240] Negotiation successful
0925 22:36:57.923118 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":149,"threads_started":1}
I20260925 22:36:57.925971 17583 heartbeater.cc:347] Connected to a master server at 127.6.246.254:35903
I20260925 22:36:57.926249 17583 heartbeater.cc:464] Registering TS with master...
I20260925 22:36:57.926815 17583 heartbeater.cc:511] Master 127.6.246.254:35903 requested a full tablet report, sending...
I20260925 22:36:57.928933 17392 ts_manager.cc:194] Registered new tserver with Master: de318803889c4ccd8bc4ef42bbcd8bce (127.6.246.193:42073)
I20260925 22:36:57.930893 17392 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.246.193@KRBTEST.COM'} at 127.6.246.193:42403
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.194.keytab.
Entry for principal kudu/127.6.246.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.194.keytab.
I20260925 22:36:57.939877 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.194:0
--local_ip_for_outbound_sockets=127.6.246.194
--webserver_interface=127.6.246.194
--webserver_port=0
--tserver_master_addrs=127.6.246.254:35903
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.194.keytab
--principal=kudu/127.6.246.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:33253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:36:58.064134 17588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:36:58.064395 17588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:36:58.064437 17588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:36:58.068579 17588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:36:58.068637 17588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:36:58.068696 17588 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.194
I20260925 22:36:58.074092 17588 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:33253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.194.keytab
--principal=kudu/127.6.246.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.246.194
--webserver_port=0
--tserver_master_addrs=127.6.246.254:35903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17588
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.194
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:36:58.075103 17588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:36:58.076154 17588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:36:58.085626 17596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:58.085762 17588 server_base.cc:1034] running on GCE node
W20260925 22:36:58.085608 17593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:58.085594 17594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:58.086385 17588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:36:58.086992 17588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:36:58.088182 17588 hybrid_clock.cc:648] HybridClock initialized: now 1790375818088132 us; error 62 us; skew 500 ppm
Sep 25 22:36:58 dist-test-slave-wjdr krb5kdc[17347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375818, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:36:58.091071 17588 init.cc:377] Logged in from keytab as kudu/127.6.246.194@KRBTEST.COM (short username kudu)
I20260925 22:36:58.091876 17588 webserver.cc:460] Webserver started at http://127.6.246.194:38953/ using document root <none> and password file <none>
I20260925 22:36:58.092404 17588 fs_manager.cc:359] Metadata directory not provided
I20260925 22:36:58.092526 17588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:36:58.092746 17588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:36:58.094803 17588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/minicluster-data/ts-1/data/instance:
uuid: "4aeec37ce9954932b1a4de6a70598e12"
format_stamp: "Formatted at 2026-09-25 22:36:58 on dist-test-slave-wjdr"
server_key: "a5ae0fd7dd8c73bc98792a3be1760914"
server_key_iv: "871fa548eddd0fe0f5c9eb2f26e28f6c"
server_key_version: "encryptionkey@0"
I20260925 22:36:58.095353 17588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/minicluster-data/ts-1/wal/instance:
uuid: "4aeec37ce9954932b1a4de6a70598e12"
format_stamp: "Formatted at 2026-09-25 22:36:58 on dist-test-slave-wjdr"
server_key: "a5ae0fd7dd8c73bc98792a3be1760914"
server_key_iv: "871fa548eddd0fe0f5c9eb2f26e28f6c"
server_key_version: "encryptionkey@0"
I20260925 22:36:58.099040 17588 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260925 22:36:58.101511 17603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:58.102751 17588 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:36:58.102869 17588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/minicluster-data/ts-1/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/minicluster-data/ts-1/wal
uuid: "4aeec37ce9954932b1a4de6a70598e12"
format_stamp: "Formatted at 2026-09-25 22:36:58 on dist-test-slave-wjdr"
server_key: "a5ae0fd7dd8c73bc98792a3be1760914"
server_key_iv: "871fa548eddd0fe0f5c9eb2f26e28f6c"
server_key_version: "encryptionkey@0"
I20260925 22:36:58.102967 17588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:36:58.113309 17588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:36:58.116110 17588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:36:58.116297 17588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:36:58.116923 17588 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:36:58.117774 17588 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:36:58.117821 17588 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:58.117862 17588 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:36:58.117913 17588 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:58.126746 17588 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.194:43785
I20260925 22:36:58.126816 17716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.194:43785 every 8 connection(s)
I20260925 22:36:58.127676 17588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
I20260925 22:36:58.136821 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 17588
I20260925 22:36:58.137012 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/minicluster-data/ts-1/wal/instance
I20260925 22:36:58.137377 7131 external_mini_cluster.cc:1428] Setting key 8f8425fdf7a65996b2530011cb5c233e
Sep 25 22:36:58 dist-test-slave-wjdr krb5kdc[17347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375818, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.194@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
I20260925 22:36:58.141980 17431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:58.129720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.194:37485 (local address 127.6.246.254:35903)
0925 22:36:58.129906 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:58.129911 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:58.130743 (+ 832us) server_negotiation.cc:407] Connection header received
0925 22:36:58.132091 (+ 1348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:58.132094 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:58.132147 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:58.132231 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:58.133421 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:58.134049 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:58.135191 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:58.135424 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:58.135507 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:58.138143 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:58.138146 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:58.138149 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:58.138172 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:58.139705 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:58.140219 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:58.140224 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:58.140226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:58.140305 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:58.140750 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:58.140755 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:58.140759 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:58.141002 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:58.141107 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:58.141597 (+ 490us) server_negotiation.cc:299] Negotiation successful
0925 22:36:58.141740 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 22:36:58.142884 17719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:58.130158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:35903 (local address 127.6.246.194:37485)
0925 22:36:58.130601 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:58.130642 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:58.131872 (+ 1230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:58.132397 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:58.132403 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:58.132813 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:58.133280 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:58.133293 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:58.134166 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:58.134169 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:58.135076 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:58.135086 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:58.135558 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:58.135561 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:58.135698 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:58.136228 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:58.136244 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:58.137964 (+ 1720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:36:58.139820 (+ 1856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:58.139825 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:58.139841 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:58.140110 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:58.140441 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:58.140447 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:58.140450 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:58.140656 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:58.141113 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:36:58.141117 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:36:58.141332 (+ 215us) client_negotiation.cc:769] Sending connection context
0925 22:36:58.141499 (+ 167us) client_negotiation.cc:240] Negotiation successful
0925 22:36:58.141741 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":109,"threads_started":1}
I20260925 22:36:58.144052 17717 heartbeater.cc:347] Connected to a master server at 127.6.246.254:35903
I20260925 22:36:58.144318 17717 heartbeater.cc:464] Registering TS with master...
I20260925 22:36:58.144958 17717 heartbeater.cc:511] Master 127.6.246.254:35903 requested a full tablet report, sending...
I20260925 22:36:58.146091 17392 ts_manager.cc:194] Registered new tserver with Master: 4aeec37ce9954932b1a4de6a70598e12 (127.6.246.194:43785)
I20260925 22:36:58.146797 17392 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.246.194@KRBTEST.COM'} at 127.6.246.194:37485
WARNING: no policy specified for kudu/127.6.246.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.195.keytab.
Entry for principal kudu/127.6.246.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.195.keytab.
I20260925 22:36:58.165060 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.195:0
--local_ip_for_outbound_sockets=127.6.246.195
--webserver_interface=127.6.246.195
--webserver_port=0
--tserver_master_addrs=127.6.246.254:35903
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.195.keytab
--principal=kudu/127.6.246.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:33253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:36:58.274134 17722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:36:58.274395 17722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:36:58.274439 17722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:36:58.278378 17722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:36:58.278430 17722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:36:58.278486 17722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.195
I20260925 22:36:58.283699 17722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:33253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.195.keytab
--principal=kudu/127.6.246.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.246.195
--webserver_port=0
--tserver_master_addrs=127.6.246.254:35903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.195
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:36:58.284669 17722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:36:58.285950 17722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:36:58.295338 17730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:36:58.295338 17727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:58.295461 17722 server_base.cc:1034] running on GCE node
W20260925 22:36:58.295364 17728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:36:58.295962 17722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:36:58.296675 17722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:36:58.297860 17722 hybrid_clock.cc:648] HybridClock initialized: now 1790375818297830 us; error 39 us; skew 500 ppm
Sep 25 22:36:58 dist-test-slave-wjdr krb5kdc[17347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375818, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:36:58.300592 17722 init.cc:377] Logged in from keytab as kudu/127.6.246.195@KRBTEST.COM (short username kudu)
I20260925 22:36:58.301424 17722 webserver.cc:460] Webserver started at http://127.6.246.195:37317/ using document root <none> and password file <none>
I20260925 22:36:58.301963 17722 fs_manager.cc:359] Metadata directory not provided
I20260925 22:36:58.302045 17722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:36:58.302258 17722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:36:58.304381 17722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/minicluster-data/ts-2/data/instance:
uuid: "3bee404c25a943cf97bbff5b359d8a6e"
format_stamp: "Formatted at 2026-09-25 22:36:58 on dist-test-slave-wjdr"
server_key: "d2892698930cf776be1d673299f569c4"
server_key_iv: "8b32d12cd9f5f540ef07e6e6e3eb08eb"
server_key_version: "encryptionkey@0"
I20260925 22:36:58.305001 17722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/minicluster-data/ts-2/wal/instance:
uuid: "3bee404c25a943cf97bbff5b359d8a6e"
format_stamp: "Formatted at 2026-09-25 22:36:58 on dist-test-slave-wjdr"
server_key: "d2892698930cf776be1d673299f569c4"
server_key_iv: "8b32d12cd9f5f540ef07e6e6e3eb08eb"
server_key_version: "encryptionkey@0"
I20260925 22:36:58.308596 17722 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:36:58.310771 17737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:58.311878 17722 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:36:58.312008 17722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/minicluster-data/ts-2/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/minicluster-data/ts-2/wal
uuid: "3bee404c25a943cf97bbff5b359d8a6e"
format_stamp: "Formatted at 2026-09-25 22:36:58 on dist-test-slave-wjdr"
server_key: "d2892698930cf776be1d673299f569c4"
server_key_iv: "8b32d12cd9f5f540ef07e6e6e3eb08eb"
server_key_version: "encryptionkey@0"
I20260925 22:36:58.312107 17722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:36:58.328640 17722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:36:58.331730 17722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:36:58.331925 17722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:36:58.332531 17722 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:36:58.333313 17722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:36:58.333379 17722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:58.333446 17722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:36:58.333489 17722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:36:58.342334 17722 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.195:41253
I20260925 22:36:58.342360 17850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.195:41253 every 8 connection(s)
I20260925 22:36:58.343183 17722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
I20260925 22:36:58.351172 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 17722
I20260925 22:36:58.351341 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/minicluster-data/ts-2/wal/instance
I20260925 22:36:58.351686 7131 external_mini_cluster.cc:1428] Setting key f8a30cb2b926dd5c94374d18b3df43ee
Sep 25 22:36:58 dist-test-slave-wjdr krb5kdc[17347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375818, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.195@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
I20260925 22:36:58.356223 17431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:58.344852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.195:57371 (local address 127.6.246.254:35903)
0925 22:36:58.344989 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:58.344993 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:58.345799 (+ 806us) server_negotiation.cc:407] Connection header received
0925 22:36:58.347192 (+ 1393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:58.347195 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:58.347252 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:58.347351 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:58.348589 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:58.349114 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:58.350211 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:58.350432 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:58.350553 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:58.353185 (+ 2632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:58.353188 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:58.353191 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:58.353216 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:58.354578 (+ 1362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:58.355003 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:58.355006 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:58.355007 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:58.355077 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:58.355344 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:58.355347 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:58.355348 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:58.355506 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:58.355579 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:58.356008 (+ 429us) server_negotiation.cc:299] Negotiation successful
0925 22:36:58.356110 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 22:36:58.357048 17853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:58.345212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:35903 (local address 127.6.246.195:57371)
0925 22:36:58.345666 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:58.345706 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:58.347000 (+ 1294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:58.347535 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:58.347542 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:58.347909 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:36:58.348382 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:58.348395 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:58.349238 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:58.349242 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:58.350086 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:58.350094 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:58.350576 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:58.350581 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:58.350755 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:58.351385 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:36:58.351407 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:36:58.353047 (+ 1640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:36:58.354671 (+ 1624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:58.354675 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:58.354685 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:58.354911 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:58.355149 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:36:58.355152 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:36:58.355153 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:36:58.355270 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:36:58.355585 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:36:58.355590 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:36:58.355802 (+ 212us) client_negotiation.cc:769] Sending connection context
0925 22:36:58.355968 (+ 166us) client_negotiation.cc:240] Negotiation successful
0925 22:36:58.356198 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":151,"threads_started":1}
I20260925 22:36:58.358106 17851 heartbeater.cc:347] Connected to a master server at 127.6.246.254:35903
I20260925 22:36:58.358320 17851 heartbeater.cc:464] Registering TS with master...
I20260925 22:36:58.358764 17851 heartbeater.cc:511] Master 127.6.246.254:35903 requested a full tablet report, sending...
I20260925 22:36:58.359674 17392 ts_manager.cc:194] Registered new tserver with Master: 3bee404c25a943cf97bbff5b359d8a6e (127.6.246.195:41253)
I20260925 22:36:58.360231 17392 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.246.195@KRBTEST.COM'} at 127.6.246.195:57371
I20260925 22:36:58.365237 7131 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:36:58.373885 17431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:58.367058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59888 (local address 127.6.246.254:35903)
0925 22:36:58.367171 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:58.367174 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:58.367376 (+ 202us) server_negotiation.cc:407] Connection header received
0925 22:36:58.367466 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:58.367468 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:58.367502 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:58.367564 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:36:58.368205 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:58.368861 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:58.369655 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:58.369851 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:58.369924 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:58.371096 (+ 1172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:36:58.371099 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:36:58.371101 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:36:58.371121 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:36:58.372509 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:58.372888 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:58.372891 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:58.372892 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:58.372976 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:36:58.373194 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:36:58.373196 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:36:58.373197 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:36:58.373385 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:36:58.373484 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:58.373656 (+ 172us) server_negotiation.cc:299] Negotiation successful
0925 22:36:58.373740 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 22:36:58.376592 17392 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59888:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:36:58.379659 17392 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:36:58.394616 17862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:58.389050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.193:42073 (local address 127.0.0.1:60852)
0925 22:36:58.389989 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:58.389998 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:58.390086 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:58.390552 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:58.390555 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:58.390570 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:36:58.390655 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:58.390661 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:58.391659 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:58.391665 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:58.392895 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:58.392905 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:58.394048 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:58.394051 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:58.394154 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:58.394208 (+ 54us) client_negotiation.cc:769] Sending connection context
0925 22:36:58.394325 (+ 117us) client_negotiation.cc:240] Negotiation successful
0925 22:36:58.394453 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":897,"mutex_wait_us":18,"thread_start_us":100,"threads_started":1}
I20260925 22:36:58.394645 17867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:58.389359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60852 (local address 127.6.246.193:42073)
0925 22:36:58.389643 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:58.389661 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:58.390089 (+ 428us) server_negotiation.cc:407] Connection header received
0925 22:36:58.390197 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:58.390204 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:58.390408 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:58.390552 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:36:58.390748 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:58.391486 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:58.393021 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:58.393919 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:58.394037 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:58.394202 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:58.394374 (+ 172us) server_negotiation.cc:299] Negotiation successful
0925 22:36:58.394488 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":111,"threads_started":1}
I20260925 22:36:58.394640 17863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:58.388387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.194:43785 (local address 127.0.0.1:39438)
0925 22:36:58.389750 (+ 1363us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:58.389762 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:58.389856 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:58.390253 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:58.390258 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:58.390280 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:36:58.390410 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:58.390417 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:58.391907 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:58.391911 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:58.393150 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:58.393160 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:58.394262 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:58.394265 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:58.394391 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:58.394414 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 22:36:58.394465 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 22:36:58.394526 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1299,"mutex_wait_us":22,"thread_start_us":137,"threads_started":1}
I20260925 22:36:58.395210 17864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:58.388420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39174 (local address 127.6.246.195:41253)
0925 22:36:58.389403 (+ 983us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:58.389415 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:58.389752 (+ 337us) server_negotiation.cc:407] Connection header received
0925 22:36:58.389841 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:58.389846 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:58.390058 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:58.390216 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:36:58.390646 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:58.391750 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:58.393636 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:58.394346 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:58.394438 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:58.394564 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:58.394968 (+ 404us) server_negotiation.cc:299] Negotiation successful
0925 22:36:58.395060 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":889,"thread_start_us":131,"threads_started":1}
I20260925 22:36:58.395612 17866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:58.388297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.195:41253 (local address 127.0.0.1:39174)
0925 22:36:58.389396 (+ 1099us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:58.389471 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:58.389605 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:58.390394 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:58.390396 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:58.390411 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:36:58.390515 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:58.390521 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:58.391907 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:58.391911 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:58.393528 (+ 1617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:58.393535 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:58.394624 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:58.394631 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:58.394730 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:58.394753 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 22:36:58.394824 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 22:36:58.394896 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":926,"mutex_wait_us":34,"thread_start_us":92,"threads_started":1}
I20260925 22:36:58.396852 17865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:58.389016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39438 (local address 127.6.246.194:43785)
0925 22:36:58.389685 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:58.389696 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:58.389845 (+ 149us) server_negotiation.cc:407] Connection header received
0925 22:36:58.389954 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:58.389958 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:58.390109 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:58.390240 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:36:58.390532 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:58.391752 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:58.393272 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:58.394118 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:58.396398 (+ 2280us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:58.396582 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:58.396631 (+ 49us) server_negotiation.cc:299] Negotiation successful
0925 22:36:58.396716 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":91,"threads_started":1}
I20260925 22:36:58.397709 17517 tablet_service.cc:1467] Processing CreateTablet for tablet 0c0e3eef6bd64942bbff825417165983 (DEFAULT_TABLE table=test-table [id=062865c77c7e446387b0d205f3d54a12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:36:58.397921 17785 tablet_service.cc:1467] Processing CreateTablet for tablet 0c0e3eef6bd64942bbff825417165983 (DEFAULT_TABLE table=test-table [id=062865c77c7e446387b0d205f3d54a12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:36:58.398739 17517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c0e3eef6bd64942bbff825417165983. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:36:58.398770 17785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c0e3eef6bd64942bbff825417165983. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:36:58.405001 17869 tablet_bootstrap.cc:492] T 0c0e3eef6bd64942bbff825417165983 P de318803889c4ccd8bc4ef42bbcd8bce: Bootstrap starting.
I20260925 22:36:58.405030 17868 tablet_bootstrap.cc:492] T 0c0e3eef6bd64942bbff825417165983 P 3bee404c25a943cf97bbff5b359d8a6e: Bootstrap starting.
I20260925 22:36:58.406790 17651 tablet_service.cc:1467] Processing CreateTablet for tablet 0c0e3eef6bd64942bbff825417165983 (DEFAULT_TABLE table=test-table [id=062865c77c7e446387b0d205f3d54a12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:36:58.407260 17869 tablet_bootstrap.cc:654] T 0c0e3eef6bd64942bbff825417165983 P de318803889c4ccd8bc4ef42bbcd8bce: Neither blocks nor log segments found. Creating new log.
I20260925 22:36:58.407733 17868 tablet_bootstrap.cc:654] T 0c0e3eef6bd64942bbff825417165983 P 3bee404c25a943cf97bbff5b359d8a6e: Neither blocks nor log segments found. Creating new log.
I20260925 22:36:58.407920 17651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c0e3eef6bd64942bbff825417165983. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:36:58.408504 17869 log.cc:824] T 0c0e3eef6bd64942bbff825417165983 P de318803889c4ccd8bc4ef42bbcd8bce: Log is configured to *not* fsync() on all Append() calls
I20260925 22:36:58.408540 17868 log.cc:824] T 0c0e3eef6bd64942bbff825417165983 P 3bee404c25a943cf97bbff5b359d8a6e: Log is configured to *not* fsync() on all Append() calls
I20260925 22:36:58.410624 17869 tablet_bootstrap.cc:492] T 0c0e3eef6bd64942bbff825417165983 P de318803889c4ccd8bc4ef42bbcd8bce: No bootstrap required, opened a new log
I20260925 22:36:58.410624 17868 tablet_bootstrap.cc:492] T 0c0e3eef6bd64942bbff825417165983 P 3bee404c25a943cf97bbff5b359d8a6e: No bootstrap required, opened a new log
I20260925 22:36:58.410813 17869 ts_tablet_manager.cc:1397] T 0c0e3eef6bd64942bbff825417165983 P de318803889c4ccd8bc4ef42bbcd8bce: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 22:36:58.410813 17868 ts_tablet_manager.cc:1397] T 0c0e3eef6bd64942bbff825417165983 P 3bee404c25a943cf97bbff5b359d8a6e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 22:36:58.414043 17872 tablet_bootstrap.cc:492] T 0c0e3eef6bd64942bbff825417165983 P 4aeec37ce9954932b1a4de6a70598e12: Bootstrap starting.
I20260925 22:36:58.414794 17868 raft_consensus.cc:348] T 0c0e3eef6bd64942bbff825417165983 P 3bee404c25a943cf97bbff5b359d8a6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bee404c25a943cf97bbff5b359d8a6e" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 41253 } } peers { permanent_uuid: "4aeec37ce9954932b1a4de6a70598e12" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 43785 } } peers { permanent_uuid: "de318803889c4ccd8bc4ef42bbcd8bce" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 42073 } }
I20260925 22:36:58.414793 17869 raft_consensus.cc:348] T 0c0e3eef6bd64942bbff825417165983 P de318803889c4ccd8bc4ef42bbcd8bce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bee404c25a943cf97bbff5b359d8a6e" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 41253 } } peers { permanent_uuid: "4aeec37ce9954932b1a4de6a70598e12" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 43785 } } peers { permanent_uuid: "de318803889c4ccd8bc4ef42bbcd8bce" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 42073 } }
I20260925 22:36:58.415064 17869 raft_consensus.cc:374] T 0c0e3eef6bd64942bbff825417165983 P de318803889c4ccd8bc4ef42bbcd8bce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:36:58.415064 17868 raft_consensus.cc:374] T 0c0e3eef6bd64942bbff825417165983 P 3bee404c25a943cf97bbff5b359d8a6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:36:58.415123 17869 raft_consensus.cc:729] T 0c0e3eef6bd64942bbff825417165983 P de318803889c4ccd8bc4ef42bbcd8bce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de318803889c4ccd8bc4ef42bbcd8bce, State: Initialized, Role: FOLLOWER
I20260925 22:36:58.415132 17868 raft_consensus.cc:729] T 0c0e3eef6bd64942bbff825417165983 P 3bee404c25a943cf97bbff5b359d8a6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bee404c25a943cf97bbff5b359d8a6e, State: Initialized, Role: FOLLOWER
I20260925 22:36:58.415611 17869 consensus_queue.cc:260] T 0c0e3eef6bd64942bbff825417165983 P de318803889c4ccd8bc4ef42bbcd8bce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bee404c25a943cf97bbff5b359d8a6e" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 41253 } } peers { permanent_uuid: "4aeec37ce9954932b1a4de6a70598e12" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 43785 } } peers { permanent_uuid: "de318803889c4ccd8bc4ef42bbcd8bce" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 42073 } }
I20260925 22:36:58.415611 17868 consensus_queue.cc:260] T 0c0e3eef6bd64942bbff825417165983 P 3bee404c25a943cf97bbff5b359d8a6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bee404c25a943cf97bbff5b359d8a6e" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 41253 } } peers { permanent_uuid: "4aeec37ce9954932b1a4de6a70598e12" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 43785 } } peers { permanent_uuid: "de318803889c4ccd8bc4ef42bbcd8bce" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 42073 } }
I20260925 22:36:58.416661 17868 ts_tablet_manager.cc:1428] T 0c0e3eef6bd64942bbff825417165983 P 3bee404c25a943cf97bbff5b359d8a6e: Time spent starting tablet: real 0.006s user 0.002s sys 0.005s
I20260925 22:36:58.416678 17869 ts_tablet_manager.cc:1428] T 0c0e3eef6bd64942bbff825417165983 P de318803889c4ccd8bc4ef42bbcd8bce: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 22:36:58.416720 17851 heartbeater.cc:503] Master 127.6.246.254:35903 was elected leader, sending a full tablet report...
I20260925 22:36:58.417306 17872 tablet_bootstrap.cc:654] T 0c0e3eef6bd64942bbff825417165983 P 4aeec37ce9954932b1a4de6a70598e12: Neither blocks nor log segments found. Creating new log.
I20260925 22:36:58.417325 17583 heartbeater.cc:503] Master 127.6.246.254:35903 was elected leader, sending a full tablet report...
I20260925 22:36:58.418200 17872 log.cc:824] T 0c0e3eef6bd64942bbff825417165983 P 4aeec37ce9954932b1a4de6a70598e12: Log is configured to *not* fsync() on all Append() calls
I20260925 22:36:58.420042 17872 tablet_bootstrap.cc:492] T 0c0e3eef6bd64942bbff825417165983 P 4aeec37ce9954932b1a4de6a70598e12: No bootstrap required, opened a new log
I20260925 22:36:58.420241 17872 ts_tablet_manager.cc:1397] T 0c0e3eef6bd64942bbff825417165983 P 4aeec37ce9954932b1a4de6a70598e12: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 22:36:58.423188 17872 raft_consensus.cc:348] T 0c0e3eef6bd64942bbff825417165983 P 4aeec37ce9954932b1a4de6a70598e12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bee404c25a943cf97bbff5b359d8a6e" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 41253 } } peers { permanent_uuid: "4aeec37ce9954932b1a4de6a70598e12" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 43785 } } peers { permanent_uuid: "de318803889c4ccd8bc4ef42bbcd8bce" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 42073 } }
I20260925 22:36:58.423421 17872 raft_consensus.cc:374] T 0c0e3eef6bd64942bbff825417165983 P 4aeec37ce9954932b1a4de6a70598e12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:36:58.423480 17872 raft_consensus.cc:729] T 0c0e3eef6bd64942bbff825417165983 P 4aeec37ce9954932b1a4de6a70598e12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aeec37ce9954932b1a4de6a70598e12, State: Initialized, Role: FOLLOWER
I20260925 22:36:58.423895 17872 consensus_queue.cc:260] T 0c0e3eef6bd64942bbff825417165983 P 4aeec37ce9954932b1a4de6a70598e12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bee404c25a943cf97bbff5b359d8a6e" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 41253 } } peers { permanent_uuid: "4aeec37ce9954932b1a4de6a70598e12" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 43785 } } peers { permanent_uuid: "de318803889c4ccd8bc4ef42bbcd8bce" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 42073 } }
I20260925 22:36:58.424562 17717 heartbeater.cc:503] Master 127.6.246.254:35903 was elected leader, sending a full tablet report...
I20260925 22:36:58.424872 17872 ts_tablet_manager.cc:1428] T 0c0e3eef6bd64942bbff825417165983 P 4aeec37ce9954932b1a4de6a70598e12: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
I20260925 22:36:58.475929 17874 raft_consensus.cc:482] T 0c0e3eef6bd64942bbff825417165983 P de318803889c4ccd8bc4ef42bbcd8bce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:36:58.476186 17874 raft_consensus.cc:504] T 0c0e3eef6bd64942bbff825417165983 P de318803889c4ccd8bc4ef42bbcd8bce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bee404c25a943cf97bbff5b359d8a6e" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 41253 } } peers { permanent_uuid: "4aeec37ce9954932b1a4de6a70598e12" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 43785 } } peers { permanent_uuid: "de318803889c4ccd8bc4ef42bbcd8bce" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 42073 } }
I20260925 22:36:58.477401 17874 leader_election.cc:288] T 0c0e3eef6bd64942bbff825417165983 P de318803889c4ccd8bc4ef42bbcd8bce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3bee404c25a943cf97bbff5b359d8a6e (127.6.246.195:41253), 4aeec37ce9954932b1a4de6a70598e12 (127.6.246.194:43785)
I20260925 22:36:58.481556 17865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:58.477901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.193:44157 (local address 127.6.246.194:43785)
0925 22:36:58.478039 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:58.478045 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:58.478186 (+ 141us) server_negotiation.cc:407] Connection header received
0925 22:36:58.478338 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:58.478341 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:58.478397 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:58.478462 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:36:58.478770 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:58.479292 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:58.480522 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:58.480906 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:58.481027 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:58.481081 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:58.481368 (+ 287us) server_negotiation.cc:299] Negotiation successful
0925 22:36:58.481438 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 22:36:58.481559 17864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:58.477783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.193:41585 (local address 127.6.246.195:41253)
0925 22:36:58.477934 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:58.477940 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:58.478236 (+ 296us) server_negotiation.cc:407] Connection header received
0925 22:36:58.478377 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:58.478379 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:58.478419 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:58.478509 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:36:58.478832 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:58.479310 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:58.480483 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:58.480896 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:58.481035 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:58.481085 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:58.481350 (+ 265us) server_negotiation.cc:299] Negotiation successful
0925 22:36:58.481423 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 22:36:58.481580 17877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:58.477728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.195:41253 (local address 127.6.246.193:41585)
0925 22:36:58.478127 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:58.478139 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:58.478254 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:58.478532 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:58.478535 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:58.478576 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:36:58.478724 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:58.478730 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:58.479433 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:58.479437 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:58.480349 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:58.480358 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:58.481025 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:58.481028 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:58.481204 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:58.481241 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 22:36:58.481320 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 22:36:58.481394 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"mutex_wait_us":54,"spinlock_wait_cycles":1664,"thread_start_us":160,"threads_started":1}
I20260925 22:36:58.481585 17878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:58.477710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.194:43785 (local address 127.6.246.193:44157)
0925 22:36:58.478089 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:58.478101 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:36:58.478229 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:36:58.478489 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:36:58.478495 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:36:58.478513 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:36:58.478657 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:58.478663 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:58.479425 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:58.479430 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:58.480357 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:36:58.480366 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:36:58.481035 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:36:58.481039 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:36:58.481217 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:58.481254 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 22:36:58.481352 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 22:36:58.481429 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"mutex_wait_us":74,"spinlock_wait_cycles":24960,"thread_start_us":117,"threads_started":1}
I20260925 22:36:58.482357 17805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c0e3eef6bd64942bbff825417165983" candidate_uuid: "de318803889c4ccd8bc4ef42bbcd8bce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bee404c25a943cf97bbff5b359d8a6e" is_pre_election: true
I20260925 22:36:58.482692 17805 raft_consensus.cc:2454] T 0c0e3eef6bd64942bbff825417165983 P 3bee404c25a943cf97bbff5b359d8a6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de318803889c4ccd8bc4ef42bbcd8bce in term 0.
I20260925 22:36:58.482735 17671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c0e3eef6bd64942bbff825417165983" candidate_uuid: "de318803889c4ccd8bc4ef42bbcd8bce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aeec37ce9954932b1a4de6a70598e12" is_pre_election: true
I20260925 22:36:58.483016 17671 raft_consensus.cc:2454] T 0c0e3eef6bd64942bbff825417165983 P 4aeec37ce9954932b1a4de6a70598e12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de318803889c4ccd8bc4ef42bbcd8bce in term 0.
I20260925 22:36:58.483520 17472 leader_election.cc:302] T 0c0e3eef6bd64942bbff825417165983 P de318803889c4ccd8bc4ef42bbcd8bce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3bee404c25a943cf97bbff5b359d8a6e, 4aeec37ce9954932b1a4de6a70598e12, de318803889c4ccd8bc4ef42bbcd8bce; no voters:
I20260925 22:36:58.483812 17874 raft_consensus.cc:2781] T 0c0e3eef6bd64942bbff825417165983 P de318803889c4ccd8bc4ef42bbcd8bce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:36:58.483922 17874 raft_consensus.cc:482] T 0c0e3eef6bd64942bbff825417165983 P de318803889c4ccd8bc4ef42bbcd8bce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:36:58.483968 17874 raft_consensus.cc:3037] T 0c0e3eef6bd64942bbff825417165983 P de318803889c4ccd8bc4ef42bbcd8bce [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:36:58.485183 17874 raft_consensus.cc:504] T 0c0e3eef6bd64942bbff825417165983 P de318803889c4ccd8bc4ef42bbcd8bce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bee404c25a943cf97bbff5b359d8a6e" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 41253 } } peers { permanent_uuid: "4aeec37ce9954932b1a4de6a70598e12" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 43785 } } peers { permanent_uuid: "de318803889c4ccd8bc4ef42bbcd8bce" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 42073 } }
I20260925 22:36:58.485531 17874 leader_election.cc:288] T 0c0e3eef6bd64942bbff825417165983 P de318803889c4ccd8bc4ef42bbcd8bce [CANDIDATE]: Term 1 election: Requested vote from peers 3bee404c25a943cf97bbff5b359d8a6e (127.6.246.195:41253), 4aeec37ce9954932b1a4de6a70598e12 (127.6.246.194:43785)
I20260925 22:36:58.485877 17671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c0e3eef6bd64942bbff825417165983" candidate_uuid: "de318803889c4ccd8bc4ef42bbcd8bce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aeec37ce9954932b1a4de6a70598e12"
I20260925 22:36:58.486034 17671 raft_consensus.cc:3037] T 0c0e3eef6bd64942bbff825417165983 P 4aeec37ce9954932b1a4de6a70598e12 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:36:58.487030 17671 raft_consensus.cc:2454] T 0c0e3eef6bd64942bbff825417165983 P 4aeec37ce9954932b1a4de6a70598e12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de318803889c4ccd8bc4ef42bbcd8bce in term 1.
I20260925 22:36:58.487296 17805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c0e3eef6bd64942bbff825417165983" candidate_uuid: "de318803889c4ccd8bc4ef42bbcd8bce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bee404c25a943cf97bbff5b359d8a6e"
I20260925 22:36:58.487332 17473 leader_election.cc:302] T 0c0e3eef6bd64942bbff825417165983 P de318803889c4ccd8bc4ef42bbcd8bce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aeec37ce9954932b1a4de6a70598e12, de318803889c4ccd8bc4ef42bbcd8bce; no voters:
I20260925 22:36:58.487497 17874 raft_consensus.cc:2781] T 0c0e3eef6bd64942bbff825417165983 P de318803889c4ccd8bc4ef42bbcd8bce [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:36:58.487444 17805 raft_consensus.cc:3037] T 0c0e3eef6bd64942bbff825417165983 P 3bee404c25a943cf97bbff5b359d8a6e [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:36:58.487730 17874 raft_consensus.cc:686] T 0c0e3eef6bd64942bbff825417165983 P de318803889c4ccd8bc4ef42bbcd8bce [term 1 LEADER]: Becoming Leader. State: Replica: de318803889c4ccd8bc4ef42bbcd8bce, State: Running, Role: LEADER
I20260925 22:36:58.488099 17874 consensus_queue.cc:237] T 0c0e3eef6bd64942bbff825417165983 P de318803889c4ccd8bc4ef42bbcd8bce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bee404c25a943cf97bbff5b359d8a6e" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 41253 } } peers { permanent_uuid: "4aeec37ce9954932b1a4de6a70598e12" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 43785 } } peers { permanent_uuid: "de318803889c4ccd8bc4ef42bbcd8bce" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 42073 } }
I20260925 22:36:58.488791 17805 raft_consensus.cc:2454] T 0c0e3eef6bd64942bbff825417165983 P 3bee404c25a943cf97bbff5b359d8a6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de318803889c4ccd8bc4ef42bbcd8bce in term 1.
I20260925 22:36:58.491547 17392 catalog_manager.cc:5495] T 0c0e3eef6bd64942bbff825417165983 P de318803889c4ccd8bc4ef42bbcd8bce reported cstate change: term changed from 0 to 1, leader changed from <none> to de318803889c4ccd8bc4ef42bbcd8bce (127.6.246.193). New cstate: current_term: 1 leader_uuid: "de318803889c4ccd8bc4ef42bbcd8bce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bee404c25a943cf97bbff5b359d8a6e" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 41253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aeec37ce9954932b1a4de6a70598e12" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 43785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de318803889c4ccd8bc4ef42bbcd8bce" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 42073 } health_report { overall_health: HEALTHY } } }
I20260925 22:36:58.515447 17867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:58.511879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60858 (local address 127.6.246.193:42073)
0925 22:36:58.512028 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 22:36:58.512032 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:36:58.512049 (+ 17us) server_negotiation.cc:407] Connection header received
0925 22:36:58.512136 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:36:58.512139 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:36:58.512204 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:36:58.512311 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:36:58.512619 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:58.513130 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:58.514004 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:36:58.514221 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:36:58.514322 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:36:58.514630 (+ 308us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:36:58.515044 (+ 414us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:36:58.515176 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 22:36:58.515233 (+ 57us) server_negotiation.cc:299] Negotiation successful
0925 22:36:58.515282 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 22:36:58.521067 17805 raft_consensus.cc:1261] T 0c0e3eef6bd64942bbff825417165983 P 3bee404c25a943cf97bbff5b359d8a6e [term 1 FOLLOWER]: Refusing update from remote peer de318803889c4ccd8bc4ef42bbcd8bce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:36:58.521070 17671 raft_consensus.cc:1261] T 0c0e3eef6bd64942bbff825417165983 P 4aeec37ce9954932b1a4de6a70598e12 [term 1 FOLLOWER]: Refusing update from remote peer de318803889c4ccd8bc4ef42bbcd8bce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:36:58.521746 17879 consensus_queue.cc:1035] T 0c0e3eef6bd64942bbff825417165983 P de318803889c4ccd8bc4ef42bbcd8bce [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aeec37ce9954932b1a4de6a70598e12" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 43785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:36:58.521984 17874 consensus_queue.cc:1035] T 0c0e3eef6bd64942bbff825417165983 P de318803889c4ccd8bc4ef42bbcd8bce [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bee404c25a943cf97bbff5b359d8a6e" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 41253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:36:58.528630 17882 mvcc.cc:205] Tried to move back new op lower bound from 7333379352655572992 to 7333379352530657280. Current Snapshot: MvccSnapshot[applied={T|T < 7333379352655572992}]
I20260925 22:36:58.529170 17883 mvcc.cc:205] Tried to move back new op lower bound from 7333379352655572992 to 7333379352530657280. Current Snapshot: MvccSnapshot[applied={T|T < 7333379352655572992}]
I20260925 22:36:58.530346 17884 mvcc.cc:205] Tried to move back new op lower bound from 7333379352655572992 to 7333379352530657280. Current Snapshot: MvccSnapshot[applied={T|T < 7333379352655572992}]
I20260925 22:36:58.536780 17392 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59888:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:36:58.536914 17392 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59888:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:36:58.539191 17392 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 02b783ccb91a4c4b8f5e9ad898cd8582: Sending DeleteTablet for 3 replicas of tablet 0c0e3eef6bd64942bbff825417165983
I20260925 22:36:58.539969 17785 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c0e3eef6bd64942bbff825417165983 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:36:58 UTC) from {username='kudu', principal='kudu/127.6.246.254@KRBTEST.COM'} at 127.0.0.1:39174
I20260925 22:36:58.539971 17651 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c0e3eef6bd64942bbff825417165983 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:36:58 UTC) from {username='kudu', principal='kudu/127.6.246.254@KRBTEST.COM'} at 127.0.0.1:39438
I20260925 22:36:58.540102 17517 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c0e3eef6bd64942bbff825417165983 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:36:58 UTC) from {username='kudu', principal='kudu/127.6.246.254@KRBTEST.COM'} at 127.0.0.1:60852
I20260925 22:36:58.540494 17892 tablet_replica.cc:330] T 0c0e3eef6bd64942bbff825417165983 P 3bee404c25a943cf97bbff5b359d8a6e: stopping tablet replica
I20260925 22:36:58.540585 17894 tablet_replica.cc:330] T 0c0e3eef6bd64942bbff825417165983 P de318803889c4ccd8bc4ef42bbcd8bce: stopping tablet replica
I20260925 22:36:58.540782 17894 raft_consensus.cc:2229] T 0c0e3eef6bd64942bbff825417165983 P de318803889c4ccd8bc4ef42bbcd8bce [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:36:58.540783 17892 raft_consensus.cc:2229] T 0c0e3eef6bd64942bbff825417165983 P 3bee404c25a943cf97bbff5b359d8a6e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:36:58.541090 17892 raft_consensus.cc:2258] T 0c0e3eef6bd64942bbff825417165983 P 3bee404c25a943cf97bbff5b359d8a6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:36:58.541141 17894 raft_consensus.cc:2258] T 0c0e3eef6bd64942bbff825417165983 P de318803889c4ccd8bc4ef42bbcd8bce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:36:58.541375 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 17454
I20260925 22:36:58.543066 17893 tablet_replica.cc:330] T 0c0e3eef6bd64942bbff825417165983 P 4aeec37ce9954932b1a4de6a70598e12: stopping tablet replica
I20260925 22:36:58.543210 17892 ts_tablet_manager.cc:1905] T 0c0e3eef6bd64942bbff825417165983 P 3bee404c25a943cf97bbff5b359d8a6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:36:58.543356 17893 raft_consensus.cc:2229] T 0c0e3eef6bd64942bbff825417165983 P 4aeec37ce9954932b1a4de6a70598e12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:36:58.543634 17893 raft_consensus.cc:2258] T 0c0e3eef6bd64942bbff825417165983 P 4aeec37ce9954932b1a4de6a70598e12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:36:58.545054 17893 ts_tablet_manager.cc:1905] T 0c0e3eef6bd64942bbff825417165983 P 4aeec37ce9954932b1a4de6a70598e12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:36:58.545749 17892 ts_tablet_manager.cc:1918] T 0c0e3eef6bd64942bbff825417165983 P 3bee404c25a943cf97bbff5b359d8a6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:36:58.545858 17892 log.cc:1196] T 0c0e3eef6bd64942bbff825417165983 P 3bee404c25a943cf97bbff5b359d8a6e: Deleting WAL directory at /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/minicluster-data/ts-2/wal/wals/0c0e3eef6bd64942bbff825417165983
I20260925 22:36:58.546187 17892 ts_tablet_manager.cc:1939] T 0c0e3eef6bd64942bbff825417165983 P 3bee404c25a943cf97bbff5b359d8a6e: Deleting consensus metadata
I20260925 22:36:58.547302 17378 catalog_manager.cc:4841] TS 3bee404c25a943cf97bbff5b359d8a6e (127.6.246.195:41253): tablet 0c0e3eef6bd64942bbff825417165983 (table test-table [id=062865c77c7e446387b0d205f3d54a12]) successfully deleted
I20260925 22:36:58.548676 17893 ts_tablet_manager.cc:1918] T 0c0e3eef6bd64942bbff825417165983 P 4aeec37ce9954932b1a4de6a70598e12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:36:58.548780 17893 log.cc:1196] T 0c0e3eef6bd64942bbff825417165983 P 4aeec37ce9954932b1a4de6a70598e12: Deleting WAL directory at /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790375752797504-7131-0/minicluster-data/ts-1/wal/wals/0c0e3eef6bd64942bbff825417165983
I20260925 22:36:58.549115 17893 ts_tablet_manager.cc:1939] T 0c0e3eef6bd64942bbff825417165983 P 4aeec37ce9954932b1a4de6a70598e12: Deleting consensus metadata
W20260925 22:36:58.549978 17377 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.246.193:42073 (error 108)
I20260925 22:36:58.550235 17379 catalog_manager.cc:4841] TS 4aeec37ce9954932b1a4de6a70598e12 (127.6.246.194:43785): tablet 0c0e3eef6bd64942bbff825417165983 (table test-table [id=062865c77c7e446387b0d205f3d54a12]) successfully deleted
I20260925 22:36:58.550750 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 17588
I20260925 22:36:58.550976 17866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:36:58.550661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.193:42073 (local address 127.0.0.1:60864)
0925 22:36:58.550797 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 22:36:58.550882 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.193:42073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260925 22:36:58.556438 17377 catalog_manager.cc:4584] TS de318803889c4ccd8bc4ef42bbcd8bce (127.6.246.193:42073): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0c0e3eef6bd64942bbff825417165983: Network error: Client connection negotiation failed: client connection to 127.6.246.193:42073: connect: Connection refused (error 111)
I20260925 22:36:58.556802 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 17722
I20260925 22:36:58.562268 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 17360
2026-09-25T22:36:58Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2336 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:36:58 dist-test-slave-wjdr krb5kdc[17898](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:36:58 dist-test-slave-wjdr krb5kdc[17898](info): set up 2 sockets
Sep 25 22:36:58 dist-test-slave-wjdr krb5kdc[17898](info): commencing operation
krb5kdc: starting...
W20260925 22:36:59.594187 7131 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Sep 25 22:36:59 dist-test-slave-wjdr krb5kdc[17898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:36:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:36:59Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.6.246.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.6.246.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.6.246.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/krb5kdc/rangeradmin_127.6.246.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.6.246.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/krb5kdc/rangeradmin_127.6.246.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.6.246.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.6.246.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.6.246.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/krb5kdc/rangerlookup_127.6.246.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.6.246.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/krb5kdc/rangerlookup_127.6.246.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.246.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.246.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.246.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/krb5kdc/HTTP_127.6.246.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.246.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/krb5kdc/HTTP_127.6.246.212@KRBTEST.COM.keytab.
I20260925 22:36:59.776050 7131 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskan1x8t/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/postgres -l logfile start
2026-09-25 22:37:02.321 UTC [17928] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 22:37:02.321 UTC [17928] LOG: listening on IPv4 address "127.6.246.212", port 33723
2026-09-25 22:37:02.325 UTC [17928] LOG: listening on Unix socket "/tmp/.s.PGSQL.33723"
2026-09-25 22:37:02.350 UTC [17931] LOG: database system was shut down at 2026-09-25 22:37:00 UTC
2026-09-25 22:37:02.354 UTC [17928] LOG: database system is ready to accept connections
I20260925 22:37:03.305348 7131 mini_postgres.cc:96] Postgres bound to 33723
2026-09-25 22:37:03.312 UTC [17939] FATAL: database "slave" does not exist
127.6.246.212:33723 - accepting connections
I20260925 22:37:03.313303 7131 mini_ranger.cc:161] Starting Ranger...
I20260925 22:37:03.328742 7131 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 22:37:03.835769 7131 mini_ranger.cc:87] Created ranger Postgres database
I20260925 22:37:03.835878 7131 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/ranger-admin
2026-09-25 22:37:04,156 [I] DB FLAVOR :POSTGRES
2026-09-25 22:37:04,157 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 22:37:04,157 [I] Checking connection..
2026-09-25 22:37:04,157 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskan1x8t/build/debug/bin/postgresql.jar:/tmp/dist-test-taskan1x8t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.246.212:33723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 22:37:04,411 [I] Checking connection passed.
2026-09-25 22:37:04,412 [I] --------- Verifying version history table ---------
2026-09-25 22:37:04,412 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskan1x8t/build/debug/bin/postgresql.jar:/tmp/dist-test-taskan1x8t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.246.212:33723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 22:37:04,676 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 22:37:04,677 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskan1x8t/build/debug/bin/postgresql.jar:/tmp/dist-test-taskan1x8t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.246.212:33723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 22:37:04,939 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 22:37:04,939 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 22:37:04,939 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskan1x8t/build/debug/bin/postgresql.jar:/tmp/dist-test-taskan1x8t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.246.212:33723/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskan1x8t/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 22:37:05.191 UTC [18003] WARNING: there is no transaction in progress
2026-09-25 22:37:05,198 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 22:37:05,198 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskan1x8t/build/debug/bin/postgresql.jar:/tmp/dist-test-taskan1x8t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.246.212:33723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 22:37:05,455 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 22:37:05,455 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 22:37:05,456 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskan1x8t/build/debug/bin/postgresql.jar:/tmp/dist-test-taskan1x8t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.246.212:33723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 22:37:05,725 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskan1x8t/build/debug/bin/postgresql.jar:/tmp/dist-test-taskan1x8t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.246.212:33723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 22:37:05,991 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskan1x8t/build/debug/bin/postgresql.jar:/tmp/dist-test-taskan1x8t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.246.212:33723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-wjdr.c.gcp-upstream.internal','N') ;"
2026-09-25 22:37:06,246 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskan1x8t/build/debug/bin/postgresql.jar:/tmp/dist-test-taskan1x8t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.246.212:33723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 22:37:06,497 [I] Table x_portal_user does not exist in database ranger
2026-09-25 22:37:06,498 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskan1x8t/build/debug/bin/postgresql.jar:/tmp/dist-test-taskan1x8t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.246.212:33723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 22:37:06,731 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 22:37:06,732 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskan1x8t/build/debug/bin/postgresql.jar:/tmp/dist-test-taskan1x8t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.246.212:33723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 22:37:06,962 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskan1x8t/build/debug/bin/postgresql.jar:/tmp/dist-test-taskan1x8t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.246.212:33723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 22:37:07,211 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 22:37:07,211 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskan1x8t/build/debug/bin/postgresql.jar:/tmp/dist-test-taskan1x8t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.246.212:33723/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskan1x8t/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 22:37:08.178 UTC [18111] WARNING: there is no transaction in progress
2026-09-25 22:37:08.196 UTC [18111] WARNING: there is no transaction in progress
2026-09-25 22:37:08.215 UTC [18111] WARNING: there is no transaction in progress
2026-09-25 22:37:08.233 UTC [18111] WARNING: there is no transaction in progress
2026-09-25 22:37:08.250 UTC [18111] WARNING: there is no transaction in progress
2026-09-25 22:37:08.269 UTC [18111] WARNING: there is no transaction in progress
2026-09-25 22:37:08.331 UTC [18111] WARNING: there is no transaction in progress
2026-09-25 22:37:08.338 UTC [18111] WARNING: there is no transaction in progress
2026-09-25 22:37:08.354 UTC [18111] WARNING: there is no transaction in progress
2026-09-25 22:37:08.365 UTC [18111] WARNING: there is no transaction in progress
2026-09-25 22:37:08.376 UTC [18111] WARNING: there is no transaction in progress
2026-09-25 22:37:08.391 UTC [18111] WARNING: there is no transaction in progress
2026-09-25 22:37:08.401 UTC [18111] WARNING: there is no transaction in progress
2026-09-25 22:37:08.408 UTC [18111] WARNING: there is no transaction in progress
2026-09-25 22:37:08.415 UTC [18111] WARNING: there is no transaction in progress
2026-09-25 22:37:08.423 UTC [18111] WARNING: there is no transaction in progress
2026-09-25 22:37:09,072 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 22:37:09,073 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskan1x8t/build/debug/bin/postgresql.jar:/tmp/dist-test-taskan1x8t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.246.212:33723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 22:37:09,316 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 22:37:09,316 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskan1x8t/build/debug/bin/postgresql.jar:/tmp/dist-test-taskan1x8t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.246.212:33723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 22:37:09,548 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 22:37:09,549 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskan1x8t/build/debug/bin/postgresql.jar:/tmp/dist-test-taskan1x8t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.246.212:33723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 22:37:09,815 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 22:37:09,815 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskan1x8t/build/debug/bin/postgresql.jar:/tmp/dist-test-taskan1x8t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.246.212:33723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 22:37:10,066 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskan1x8t/build/debug/bin/postgresql.jar:/tmp/dist-test-taskan1x8t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.246.212:33723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 22:37:10,323 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskan1x8t/build/debug/bin/postgresql.jar:/tmp/dist-test-taskan1x8t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.246.212:33723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-wjdr.c.gcp-upstream.internal';"
2026-09-25 22:37:10,547 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 22:37:10,547 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskan1x8t/build/debug/bin/postgresql.jar:/tmp/dist-test-taskan1x8t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.246.212:33723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 22:37:10,777 [I] DB_PATCHES have already been applied
I20260925 22:37:10.783012 7131 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/ranger-admin:/tmp/dist-test-taskan1x8t/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskan1x8t/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskan1x8t/build/debug/bin/postgresql.jar:/tmp/dist-test-taskan1x8t/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 22:37:10.783093 7131 mini_ranger.cc:193] Using host: 127.6.246.212
I20260925 22:37:10.786882 7131 mini_ranger.cc:227] Ranger admin URL: http://127.6.246.212:41029
Sep 25, 2026 10:37:11 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 10:37:11 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 10:37:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 10:37:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 10:37:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 10:37:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 10:37:11 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 22:37:11.887181 7131 mini_ranger.cc:160] Time spent starting Ranger: real 8.574s user 0.000s sys 0.010s
Sep 25, 2026 10:37:32 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 10:37:32 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Sep 25, 2026 10:37:32 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 10:37:32 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 10:37:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 10:37:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 10:37:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 10:37:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 10:37:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 10:37:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 10:37:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 10:37:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 10:37:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 10:37:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 10:37:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 10:37:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 10:37:32 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 10:37:33 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 22:37:34.061941 7131 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.6.246.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.6.246.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.6.246.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/krb5kdc/rangerkms_127.6.246.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.6.246.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/krb5kdc/rangerkms_127.6.246.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.246.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.6.246.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.246.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/krb5kdc/HTTP_127.6.246.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.246.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/krb5kdc/HTTP_127.6.246.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 22:37:34 dist-test-slave-wjdr krb5kdc[17898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375854, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 22:37:34.159906 7131 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 22:37:34.170997 7131 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 22:37:35.073336 7131 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 22:37:35.073444 7131 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/ranger-kms
I20260925 22:37:35.073585 7131 mini_ranger_kms.cc:227] Using postgres at 127.6.246.212:33723
2026-09-25 22:37:35,316 [I] DB FLAVOR :POSTGRES
2026-09-25 22:37:35,316 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 22:37:35,317 [I] Checking connection
2026-09-25 22:37:35,317 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskan1x8t/build/debug/bin/postgresql.jar:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.246.212:33723/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 22:37:35,592 [I] connection success
2026-09-25 22:37:35,592 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 22:37:35,592 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 22:37:35,593 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskan1x8t/build/debug/bin/postgresql.jar:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.246.212:33723/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 22:37:35,857 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 22:37:35,857 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 22:37:35,857 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 22:37:35,858 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskan1x8t/build/debug/bin/postgresql.jar:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.246.212:33723/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 22:37:36,157 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 22:37:36.658959 7131 mini_ranger_kms.cc:310] Created kms service
I20260925 22:37:36.846807 7131 mini_ranger_kms.cc:326] Created kudu user
I20260925 22:37:36.896716 7131 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 22:37:36 dist-test-slave-wjdr krb5kdc[17898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375854, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.6.246.212@KRBTEST.COM
I20260925 22:37:37.300024 7131 mini_ranger_kms.cc:384] Added ranger policy
I20260925 22:37:37.300190 7131 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/ranger-kms:/tmp/dist-test-taskan1x8t/build/debug/bin/postgresql.jar:/tmp/dist-test-taskan1x8t/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskan1x8t/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 22:37:37.300213 7131 mini_ranger_kms.cc:240] Using host: 127.6.246.212
I20260925 22:37:37.304358 7131 mini_ranger_kms.cc:276] Ranger KMS PID: 18291
I20260925 22:37:37.304558 7131 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.6.246.212:33987
Sep 25, 2026 10:37:38 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 10:37:38 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 10:37:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 10:37:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 10:37:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 10:37:39 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260925 22:37:39.588443 7131 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.429s user 0.000s sys 0.226s
I20260925 22:37:39.588605 7131 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 22:37:39.588725 7131 mini_ranger_kms.cc:401] 127.6.246.212:33987/kms/v1/keys
Sep 25 22:37:39 dist-test-slave-wjdr krb5kdc[17898](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790375859, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.246.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 25 22:37:40 dist-test-slave-wjdr krb5kdc[17898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375859, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.246.212@KRBTEST.COM for HTTP/127.6.246.212@KRBTEST.COM
Sep 25 22:37:40 dist-test-slave-wjdr krb5kdc[17898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375859, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.246.212@KRBTEST.COM for HTTP/127.6.246.212@KRBTEST.COM
Sep 25 22:37:42 dist-test-slave-wjdr krb5kdc[17898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.6.246.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab.
Entry for principal kudu/127.6.246.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab.
I20260925 22:37:42.739910 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:40061
--webserver_interface=127.6.246.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab
--principal=kudu/127.6.246.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:32865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.246.254:40061
--ranger_config_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.246.212:33987/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:37:42.850317 18342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:37:42.850572 18342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:37:42.850610 18342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:37:42.854627 18342 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:37:42.854677 18342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:37:42.854696 18342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:37:42.854738 18342 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:37:42.854770 18342 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:37:42.860092 18342 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:32865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/master-0/wal
--ranger_kms_url=127.6.246.212:33987/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.6.246.254:40061
--ranger_config_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab
--principal=kudu/127.6.246.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:40061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.246.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:37:42.861080 18342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:37:42.862062 18342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:37:42.870874 18348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:37:42.870957 18347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:37:42.870944 18342 server_base.cc:1034] running on GCE node
W20260925 22:37:42.870882 18350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:37:42.871680 18342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:37:42.872807 18342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:37:42.873987 18342 hybrid_clock.cc:648] HybridClock initialized: now 1790375862873962 us; error 38 us; skew 500 ppm
Sep 25 22:37:42 dist-test-slave-wjdr krb5kdc[17898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375862, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:37:42.881656 18342 init.cc:377] Logged in from keytab as kudu/127.6.246.254@KRBTEST.COM (short username kudu)
I20260925 22:37:42.882536 18342 webserver.cc:460] Webserver started at http://127.6.246.254:41857/ using document root <none> and password file <none>
I20260925 22:37:42.883078 18342 fs_manager.cc:359] Metadata directory not provided
I20260925 22:37:42.883150 18342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:37:42.883363 18342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 22:37:42 dist-test-slave-wjdr krb5kdc[17898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375862, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.254@KRBTEST.COM for HTTP/127.6.246.212@KRBTEST.COM
I20260925 22:37:42.937145 18342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/master-0/data/instance:
uuid: "8608c8d900de46cdb8fe5ae31c2e9d4b"
format_stamp: "Formatted at 2026-09-25 22:37:42 on dist-test-slave-wjdr"
server_key: "bc3e0e4327cfd1443bc1c1a882b2a916"
server_key_iv: "0153e15638a2da14d0c4415133ecb80a"
server_key_version: "kuduclusterkey@0"
I20260925 22:37:42.937863 18342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/master-0/wal/instance:
uuid: "8608c8d900de46cdb8fe5ae31c2e9d4b"
format_stamp: "Formatted at 2026-09-25 22:37:42 on dist-test-slave-wjdr"
server_key: "bc3e0e4327cfd1443bc1c1a882b2a916"
server_key_iv: "0153e15638a2da14d0c4415133ecb80a"
server_key_version: "kuduclusterkey@0"
I20260925 22:37:42.942809 18342 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 22:37:42.956358 18357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:37:42.957748 18342 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 22:37:42.957886 18342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/master-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/master-0/wal
uuid: "8608c8d900de46cdb8fe5ae31c2e9d4b"
format_stamp: "Formatted at 2026-09-25 22:37:42 on dist-test-slave-wjdr"
server_key: "bc3e0e4327cfd1443bc1c1a882b2a916"
server_key_iv: "0153e15638a2da14d0c4415133ecb80a"
server_key_version: "kuduclusterkey@0"
I20260925 22:37:42.958029 18342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:37:42.982972 18342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:37:42.986130 18342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:37:42.986347 18342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:37:42.994525 18342 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.254:40061
I20260925 22:37:42.994539 18409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.254:40061 every 8 connection(s)
I20260925 22:37:42.995478 18342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
I20260925 22:37:42.997056 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 18342
I20260925 22:37:42.997203 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/master-0/wal/instance
I20260925 22:37:42.997505 7131 external_mini_cluster.cc:1428] Setting key 961424690de5fb6e11ebeb82a898833c
I20260925 22:37:42.998920 18410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:37:43.005769 18410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8608c8d900de46cdb8fe5ae31c2e9d4b: Bootstrap starting.
Sep 25 22:37:43 dist-test-slave-wjdr krb5kdc[17898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
I20260925 22:37:43.008622 18410 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8608c8d900de46cdb8fe5ae31c2e9d4b: Neither blocks nor log segments found. Creating new log.
I20260925 22:37:43.009999 18410 log.cc:824] T 00000000000000000000000000000000 P 8608c8d900de46cdb8fe5ae31c2e9d4b: Log is configured to *not* fsync() on all Append() calls
I20260925 22:37:43.012104 18413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:37:42.998798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59584 (local address 127.6.246.254:40061)
0925 22:37:42.999220 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0925 22:37:42.999240 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 22:37:42.999433 (+ 193us) server_negotiation.cc:407] Connection header received
0925 22:37:43.000208 (+ 775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:37:43.000223 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:37:43.000608 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:37:43.000920 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:37:43.001699 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:37:43.003098 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:37:43.003953 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:37:43.004417 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:37:43.004638 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:37:43.007269 (+ 2631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:37:43.007274 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:37:43.007286 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:37:43.007314 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:37:43.009113 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:37:43.009542 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:37:43.009547 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:37:43.009552 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:37:43.009653 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:37:43.009925 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:37:43.009930 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:37:43.009932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:37:43.010375 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:37:43.010577 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0925 22:37:43.011029 (+ 452us) server_negotiation.cc:299] Negotiation successful
0925 22:37:43.011420 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":138,"threads_started":1}
I20260925 22:37:43.012338 18410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8608c8d900de46cdb8fe5ae31c2e9d4b: No bootstrap required, opened a new log
I20260925 22:37:43.016328 18410 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8608c8d900de46cdb8fe5ae31c2e9d4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8608c8d900de46cdb8fe5ae31c2e9d4b" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 40061 } }
I20260925 22:37:43.016628 18410 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8608c8d900de46cdb8fe5ae31c2e9d4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:37:43.016692 18410 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8608c8d900de46cdb8fe5ae31c2e9d4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8608c8d900de46cdb8fe5ae31c2e9d4b, State: Initialized, Role: FOLLOWER
I20260925 22:37:43.017138 18410 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8608c8d900de46cdb8fe5ae31c2e9d4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8608c8d900de46cdb8fe5ae31c2e9d4b" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 40061 } }
I20260925 22:37:43.017293 18410 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8608c8d900de46cdb8fe5ae31c2e9d4b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:37:43.017354 18410 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8608c8d900de46cdb8fe5ae31c2e9d4b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:37:43.017442 18410 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8608c8d900de46cdb8fe5ae31c2e9d4b [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:37:43.018528 18410 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8608c8d900de46cdb8fe5ae31c2e9d4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8608c8d900de46cdb8fe5ae31c2e9d4b" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 40061 } }
I20260925 22:37:43.018872 18410 leader_election.cc:302] T 00000000000000000000000000000000 P 8608c8d900de46cdb8fe5ae31c2e9d4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8608c8d900de46cdb8fe5ae31c2e9d4b; no voters:
I20260925 22:37:43.019194 18410 leader_election.cc:288] T 00000000000000000000000000000000 P 8608c8d900de46cdb8fe5ae31c2e9d4b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:37:43.019296 18415 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8608c8d900de46cdb8fe5ae31c2e9d4b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:37:43.019536 18415 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8608c8d900de46cdb8fe5ae31c2e9d4b [term 1 LEADER]: Becoming Leader. State: Replica: 8608c8d900de46cdb8fe5ae31c2e9d4b, State: Running, Role: LEADER
I20260925 22:37:43.019843 18415 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8608c8d900de46cdb8fe5ae31c2e9d4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8608c8d900de46cdb8fe5ae31c2e9d4b" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 40061 } }
I20260925 22:37:43.020495 18410 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8608c8d900de46cdb8fe5ae31c2e9d4b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:37:43.021330 18416 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8608c8d900de46cdb8fe5ae31c2e9d4b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8608c8d900de46cdb8fe5ae31c2e9d4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8608c8d900de46cdb8fe5ae31c2e9d4b" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 40061 } } }
I20260925 22:37:43.021458 18416 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8608c8d900de46cdb8fe5ae31c2e9d4b [sys.catalog]: This master's current role is: LEADER
I20260925 22:37:43.021693 18416 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8608c8d900de46cdb8fe5ae31c2e9d4b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8608c8d900de46cdb8fe5ae31c2e9d4b. Latest consensus state: current_term: 1 leader_uuid: "8608c8d900de46cdb8fe5ae31c2e9d4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8608c8d900de46cdb8fe5ae31c2e9d4b" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 40061 } } }
I20260925 22:37:43.021775 18416 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8608c8d900de46cdb8fe5ae31c2e9d4b [sys.catalog]: This master's current role is: LEADER
I20260925 22:37:43.022708 18410 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 22:37:43.022977 18424 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:37:43.029479 18424 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:37:43.038221 18424 catalog_manager.cc:1347] Generated new cluster ID: f9b7ee07897041d3bc45e18afa6342f7
I20260925 22:37:43.038293 18424 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:37:43.062103 18424 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:37:43.063352 18424 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:37:43.071370 18424 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8608c8d900de46cdb8fe5ae31c2e9d4b: Generated new TSK 0
I20260925 22:37:43.072216 18424 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.246.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.193.keytab.
Entry for principal kudu/127.6.246.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.193.keytab.
I20260925 22:37:43.785998 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.246.212:33987/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.193:0
--local_ip_for_outbound_sockets=127.6.246.193
--webserver_interface=127.6.246.193
--webserver_port=0
--tserver_master_addrs=127.6.246.254:40061
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.193.keytab
--principal=kudu/127.6.246.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:32865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:37:43.934716 18455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:37:43.935062 18455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:37:43.935154 18455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:37:43.941214 18455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:37:43.941298 18455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:37:43.941397 18455 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.193
I20260925 22:37:43.949069 18455 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:32865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.6.246.212:33987/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.193.keytab
--principal=kudu/127.6.246.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.246.193
--webserver_port=0
--tserver_master_addrs=127.6.246.254:40061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.193
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:37:43.950555 18455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:37:43.951877 18455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:37:43.963836 18462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:37:43.964067 18465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:37:43.964359 18463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:37:43.965073 18455 server_base.cc:1034] running on GCE node
I20260925 22:37:43.965561 18455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:37:43.966341 18455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:37:43.967542 18455 hybrid_clock.cc:648] HybridClock initialized: now 1790375863967520 us; error 35 us; skew 500 ppm
Sep 25 22:37:43 dist-test-slave-wjdr krb5kdc[17898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375863, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:37:43.971009 18455 init.cc:377] Logged in from keytab as kudu/127.6.246.193@KRBTEST.COM (short username kudu)
I20260925 22:37:43.971884 18455 webserver.cc:460] Webserver started at http://127.6.246.193:46327/ using document root <none> and password file <none>
I20260925 22:37:43.972440 18455 fs_manager.cc:359] Metadata directory not provided
I20260925 22:37:43.972569 18455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:37:43.972805 18455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 22:37:43 dist-test-slave-wjdr krb5kdc[17898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375863, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.193@KRBTEST.COM for HTTP/127.6.246.212@KRBTEST.COM
I20260925 22:37:43.986793 18455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/ts-0/data/instance:
uuid: "d3a2f3bb98894f778343ad0985002b87"
format_stamp: "Formatted at 2026-09-25 22:37:43 on dist-test-slave-wjdr"
server_key: "ea96ac245735e3dfc52dc847deff2d42"
server_key_iv: "9e1f60909e7531601e21879d6f82421a"
server_key_version: "kuduclusterkey@0"
I20260925 22:37:43.987613 18455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/ts-0/wal/instance:
uuid: "d3a2f3bb98894f778343ad0985002b87"
format_stamp: "Formatted at 2026-09-25 22:37:43 on dist-test-slave-wjdr"
server_key: "ea96ac245735e3dfc52dc847deff2d42"
server_key_iv: "9e1f60909e7531601e21879d6f82421a"
server_key_version: "kuduclusterkey@0"
I20260925 22:37:44.014451 18455 fs_manager.cc:691] Time spent creating directory manager: real 0.026s user 0.004s sys 0.000s
I20260925 22:37:44.034653 18474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:37:44.036126 18455 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:37:44.036269 18455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/ts-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/ts-0/wal
uuid: "d3a2f3bb98894f778343ad0985002b87"
format_stamp: "Formatted at 2026-09-25 22:37:43 on dist-test-slave-wjdr"
server_key: "ea96ac245735e3dfc52dc847deff2d42"
server_key_iv: "9e1f60909e7531601e21879d6f82421a"
server_key_version: "kuduclusterkey@0"
I20260925 22:37:44.036412 18455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:37:44.072256 18455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:37:44.076210 18455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:37:44.076498 18455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:37:44.077262 18455 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:37:44.078356 18455 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:37:44.078464 18455 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:37:44.078590 18455 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:37:44.078667 18455 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:37:44.093894 18455 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.193:43189
I20260925 22:37:44.093899 18587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.193:43189 every 8 connection(s)
I20260925 22:37:44.095192 18455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
I20260925 22:37:44.099175 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 18455
I20260925 22:37:44.099308 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/ts-0/wal/instance
I20260925 22:37:44.103571 7131 external_mini_cluster.cc:1428] Setting key c0bc860e7d1fc9f5ef07e26df4d50768
Sep 25 22:37:44 dist-test-slave-wjdr krb5kdc[17898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375863, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.193@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.246.194@KRBTEST.COM; defaulting to no policy
I20260925 22:37:44.118754 18590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:37:44.097890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:40061 (local address 127.6.246.193:45091)
0925 22:37:44.099243 (+ 1353us) negotiation.cc:107] Waiting for socket to connect
0925 22:37:44.099309 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0925 22:37:44.100519 (+ 1210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:37:44.105077 (+ 4558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:37:44.105087 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:37:44.105708 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:37:44.106459 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:37:44.106482 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:37:44.107667 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:37:44.107672 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:37:44.109167 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:37:44.109180 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:37:44.109927 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:37:44.109933 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:37:44.110156 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:37:44.110859 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:37:44.110883 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:37:44.112742 (+ 1859us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:37:44.115156 (+ 2414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:37:44.115164 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:37:44.115183 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:37:44.115464 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:37:44.115792 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:37:44.115798 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:37:44.115800 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:37:44.116007 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:37:44.116566 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:37:44.116574 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:37:44.116985 (+ 411us) client_negotiation.cc:769] Sending connection context
0925 22:37:44.117211 (+ 226us) client_negotiation.cc:240] Negotiation successful
0925 22:37:44.117562 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1114,"thread_start_us":139,"threads_started":1}
I20260925 22:37:44.119902 18591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:37:44.102889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.193:45091 (local address 127.6.246.254:40061)
0925 22:37:44.104622 (+ 1733us) server_negotiation.cc:206] Beginning negotiation
0925 22:37:44.104627 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:37:44.104640 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:37:44.104677 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:37:44.104680 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:37:44.104729 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:37:44.106623 (+ 1894us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:37:44.106737 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:37:44.107526 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:37:44.109351 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:37:44.109743 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:37:44.110527 (+ 784us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:37:44.112926 (+ 2399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:37:44.112933 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:37:44.112939 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:37:44.112973 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:37:44.115045 (+ 2072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:37:44.115579 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:37:44.115583 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:37:44.115585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:37:44.115680 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:37:44.116130 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:37:44.116133 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:37:44.116135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:37:44.116383 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:37:44.116505 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 22:37:44.119584 (+ 3079us) server_negotiation.cc:299] Negotiation successful
0925 22:37:44.119742 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1649,"thread_start_us":94,"threads_started":1}
I20260925 22:37:44.121687 18588 heartbeater.cc:347] Connected to a master server at 127.6.246.254:40061
I20260925 22:37:44.121977 18588 heartbeater.cc:464] Registering TS with master...
I20260925 22:37:44.122658 18588 heartbeater.cc:511] Master 127.6.246.254:40061 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.194@KRBTEST.COM" created.
I20260925 22:37:44.124430 18374 ts_manager.cc:194] Registered new tserver with Master: d3a2f3bb98894f778343ad0985002b87 (127.6.246.193:43189)
I20260925 22:37:44.125782 18374 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.246.193@KRBTEST.COM'} at 127.6.246.193:45091
Sep 25 22:37:44 dist-test-slave-wjdr krb5kdc[17898](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790375864, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.194.keytab.
Entry for principal kudu/127.6.246.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.194.keytab.
I20260925 22:37:44.147360 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.246.212:33987/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.194:0
--local_ip_for_outbound_sockets=127.6.246.194
--webserver_interface=127.6.246.194
--webserver_port=0
--tserver_master_addrs=127.6.246.254:40061
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.194.keytab
--principal=kudu/127.6.246.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:32865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:37:44.275933 18594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:37:44.276229 18594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:37:44.276310 18594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:37:44.280567 18594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:37:44.280668 18594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:37:44.280783 18594 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.194
I20260925 22:37:44.286398 18594 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:32865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.6.246.212:33987/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.194.keytab
--principal=kudu/127.6.246.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.246.194
--webserver_port=0
--tserver_master_addrs=127.6.246.254:40061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18594
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.194
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:37:44.287592 18594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:37:44.288801 18594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:37:44.300145 18600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:37:44.300144 18603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:37:44.300504 18601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:37:44.300925 18594 server_base.cc:1034] running on GCE node
I20260925 22:37:44.301283 18594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:37:44.301910 18594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:37:44.303089 18594 hybrid_clock.cc:648] HybridClock initialized: now 1790375864303061 us; error 51 us; skew 500 ppm
Sep 25 22:37:44 dist-test-slave-wjdr krb5kdc[17898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375864, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:37:44.306849 18594 init.cc:377] Logged in from keytab as kudu/127.6.246.194@KRBTEST.COM (short username kudu)
I20260925 22:37:44.308058 18594 webserver.cc:460] Webserver started at http://127.6.246.194:41223/ using document root <none> and password file <none>
I20260925 22:37:44.308828 18594 fs_manager.cc:359] Metadata directory not provided
I20260925 22:37:44.308959 18594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:37:44.309222 18594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 22:37:44 dist-test-slave-wjdr krb5kdc[17898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375864, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.194@KRBTEST.COM for HTTP/127.6.246.212@KRBTEST.COM
I20260925 22:37:44.328683 18594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/ts-1/data/instance:
uuid: "884a72306b8b4334b0b1f4e2f491be15"
format_stamp: "Formatted at 2026-09-25 22:37:44 on dist-test-slave-wjdr"
server_key: "d01fbb3926acf9f3ce33b9e86b2c9c9b"
server_key_iv: "fa42c536a4c9b543a48bb8228422fe62"
server_key_version: "kuduclusterkey@0"
I20260925 22:37:44.329448 18594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/ts-1/wal/instance:
uuid: "884a72306b8b4334b0b1f4e2f491be15"
format_stamp: "Formatted at 2026-09-25 22:37:44 on dist-test-slave-wjdr"
server_key: "d01fbb3926acf9f3ce33b9e86b2c9c9b"
server_key_iv: "fa42c536a4c9b543a48bb8228422fe62"
server_key_version: "kuduclusterkey@0"
I20260925 22:37:44.334714 18594 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 22:37:44.358434 18610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:37:44.360276 18594 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:37:44.360415 18594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/ts-1/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/ts-1/wal
uuid: "884a72306b8b4334b0b1f4e2f491be15"
format_stamp: "Formatted at 2026-09-25 22:37:44 on dist-test-slave-wjdr"
server_key: "d01fbb3926acf9f3ce33b9e86b2c9c9b"
server_key_iv: "fa42c536a4c9b543a48bb8228422fe62"
server_key_version: "kuduclusterkey@0"
I20260925 22:37:44.360594 18594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:37:44.387300 18594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:37:44.391077 18594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:37:44.391281 18594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:37:44.391999 18594 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:37:44.393040 18594 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:37:44.393105 18594 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:37:44.393167 18594 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:37:44.393196 18594 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:37:44.406669 18594 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.194:41483
I20260925 22:37:44.407797 18594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
I20260925 22:37:44.410115 18723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.194:41483 every 8 connection(s)
I20260925 22:37:44.415794 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 18594
I20260925 22:37:44.415920 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/ts-1/wal/instance
I20260925 22:37:44.416201 7131 external_mini_cluster.cc:1428] Setting key fa3591130c86d3d9e41993c24106b6b1
Sep 25 22:37:44 dist-test-slave-wjdr krb5kdc[17898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375864, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.194@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
I20260925 22:37:44.426709 18591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:37:44.411775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.194:35263 (local address 127.6.246.254:40061)
0925 22:37:44.411918 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 22:37:44.411923 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:37:44.413009 (+ 1086us) server_negotiation.cc:407] Connection header received
0925 22:37:44.414222 (+ 1213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:37:44.414226 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:37:44.414282 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:37:44.414376 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:37:44.415700 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:37:44.416527 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:37:44.418351 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:37:44.418722 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:37:44.418797 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:37:44.422526 (+ 3729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:37:44.422530 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:37:44.422533 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:37:44.422560 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:37:44.424279 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:37:44.424947 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:37:44.424952 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:37:44.424954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:37:44.425049 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:37:44.425455 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:37:44.425458 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:37:44.425460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:37:44.425676 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:37:44.425776 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 22:37:44.426400 (+ 624us) server_negotiation.cc:299] Negotiation successful
0925 22:37:44.426537 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 22:37:44.427649 18726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:37:44.412265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:40061 (local address 127.6.246.194:35263)
0925 22:37:44.412836 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0925 22:37:44.412904 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0925 22:37:44.413969 (+ 1065us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:37:44.414559 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:37:44.414567 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:37:44.414907 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:37:44.415521 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:37:44.415539 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:37:44.416811 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:37:44.416814 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:37:44.418202 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:37:44.418214 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:37:44.418996 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:37:44.418999 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:37:44.419190 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:37:44.420015 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:37:44.420035 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:37:44.422350 (+ 2315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:37:44.424404 (+ 2054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:37:44.424410 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:37:44.424426 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:37:44.424832 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:37:44.425156 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:37:44.425159 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:37:44.425161 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:37:44.425343 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:37:44.425790 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:37:44.425797 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:37:44.426110 (+ 313us) client_negotiation.cc:769] Sending connection context
0925 22:37:44.426312 (+ 202us) client_negotiation.cc:240] Negotiation successful
0925 22:37:44.426644 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":185,"threads_started":1}
I20260925 22:37:44.428921 18724 heartbeater.cc:347] Connected to a master server at 127.6.246.254:40061
I20260925 22:37:44.429551 18724 heartbeater.cc:464] Registering TS with master...
I20260925 22:37:44.430132 18724 heartbeater.cc:511] Master 127.6.246.254:40061 requested a full tablet report, sending...
I20260925 22:37:44.431306 18374 ts_manager.cc:194] Registered new tserver with Master: 884a72306b8b4334b0b1f4e2f491be15 (127.6.246.194:41483)
I20260925 22:37:44.432003 18374 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.246.194@KRBTEST.COM'} at 127.6.246.194:35263
WARNING: no policy specified for kudu/127.6.246.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.195.keytab.
Entry for principal kudu/127.6.246.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.195.keytab.
I20260925 22:37:44.469664 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.246.212:33987/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.195:0
--local_ip_for_outbound_sockets=127.6.246.195
--webserver_interface=127.6.246.195
--webserver_port=0
--tserver_master_addrs=127.6.246.254:40061
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.195.keytab
--principal=kudu/127.6.246.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:32865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 22:37:44 dist-test-slave-wjdr krb5kdc[17898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375864, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.254@KRBTEST.COM for HTTP/127.6.246.212@KRBTEST.COM
W20260925 22:37:44.609756 18729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:37:44.610100 18729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:37:44.610180 18729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:37:44.615458 18729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:37:44.615535 18729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:37:44.615630 18729 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.195
I20260925 22:37:44.623375 18729 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:32865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.6.246.212:33987/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.195.keytab
--principal=kudu/127.6.246.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.246.195
--webserver_port=0
--tserver_master_addrs=127.6.246.254:40061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18729
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.195
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:37:44.624724 18729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:37:44.626021 18729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:37:44.641637 18737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:37:44.642009 18729 server_base.cc:1034] running on GCE node
W20260925 22:37:44.642237 18734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:37:44.642766 18735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:37:44.643270 18729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:37:44.644069 18729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:37:44.651687 18729 hybrid_clock.cc:648] HybridClock initialized: now 1790375864651603 us; error 103 us; skew 500 ppm
Sep 25 22:37:44 dist-test-slave-wjdr krb5kdc[17898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375864, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:37:44.655652 18729 init.cc:377] Logged in from keytab as kudu/127.6.246.195@KRBTEST.COM (short username kudu)
I20260925 22:37:44.660596 18729 webserver.cc:460] Webserver started at http://127.6.246.195:42827/ using document root <none> and password file <none>
I20260925 22:37:44.661403 18729 fs_manager.cc:359] Metadata directory not provided
I20260925 22:37:44.661505 18729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:37:44.661756 18729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 22:37:44 dist-test-slave-wjdr krb5kdc[17898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375864, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.195@KRBTEST.COM for HTTP/127.6.246.212@KRBTEST.COM
I20260925 22:37:44.677385 18729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/ts-2/data/instance:
uuid: "e1d8afb6c0d1493f83eb7f1ccfdcb062"
format_stamp: "Formatted at 2026-09-25 22:37:44 on dist-test-slave-wjdr"
server_key: "a815686d81857a204d562a6a7d8d8c59"
server_key_iv: "fe5221f6c0fbe7897f6b8ec4b158a90b"
server_key_version: "kuduclusterkey@0"
I20260925 22:37:44.678324 18729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/ts-2/wal/instance:
uuid: "e1d8afb6c0d1493f83eb7f1ccfdcb062"
format_stamp: "Formatted at 2026-09-25 22:37:44 on dist-test-slave-wjdr"
server_key: "a815686d81857a204d562a6a7d8d8c59"
server_key_iv: "fe5221f6c0fbe7897f6b8ec4b158a90b"
server_key_version: "kuduclusterkey@0"
I20260925 22:37:44.685369 18729 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.000s
I20260925 22:37:44.708935 18744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:37:44.710708 18729 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.004s sys 0.000s
I20260925 22:37:44.710855 18729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/ts-2/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/ts-2/wal
uuid: "e1d8afb6c0d1493f83eb7f1ccfdcb062"
format_stamp: "Formatted at 2026-09-25 22:37:44 on dist-test-slave-wjdr"
server_key: "a815686d81857a204d562a6a7d8d8c59"
server_key_iv: "fe5221f6c0fbe7897f6b8ec4b158a90b"
server_key_version: "kuduclusterkey@0"
I20260925 22:37:44.711015 18729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:37:44.726873 18729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:37:44.729873 18729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:37:44.730070 18729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:37:44.730669 18729 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:37:44.731504 18729 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:37:44.731571 18729 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:37:44.731638 18729 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:37:44.731683 18729 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:37:44.742282 18729 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.195:34385
I20260925 22:37:44.742350 18857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.195:34385 every 8 connection(s)
I20260925 22:37:44.743243 18729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
I20260925 22:37:44.752882 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 18729
I20260925 22:37:44.753010 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/ts-2/wal/instance
I20260925 22:37:44.753320 7131 external_mini_cluster.cc:1428] Setting key 823f4247abaf500a677c004057a7a673
Sep 25 22:37:44 dist-test-slave-wjdr krb5kdc[17898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375864, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.195@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
I20260925 22:37:44.758068 18591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:37:44.745469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.195:48357 (local address 127.6.246.254:40061)
0925 22:37:44.745598 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 22:37:44.745602 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:37:44.746615 (+ 1013us) server_negotiation.cc:407] Connection header received
0925 22:37:44.748133 (+ 1518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:37:44.748137 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:37:44.748196 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:37:44.748269 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:37:44.749398 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:37:44.749928 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:37:44.751029 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:37:44.751270 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:37:44.751394 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:37:44.754439 (+ 3045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:37:44.754445 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:37:44.754451 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:37:44.754485 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:37:44.756078 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:37:44.756583 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:37:44.756586 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:37:44.756588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:37:44.756672 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:37:44.757016 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:37:44.757019 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:37:44.757021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:37:44.757202 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:37:44.757306 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 22:37:44.757785 (+ 479us) server_negotiation.cc:299] Negotiation successful
0925 22:37:44.757901 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 22:37:44.758997 18860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:37:44.745906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:40061 (local address 127.6.246.195:48357)
0925 22:37:44.746421 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0925 22:37:44.746518 (+ 97us) client_negotiation.cc:174] Beginning negotiation
0925 22:37:44.747917 (+ 1399us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:37:44.748434 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:37:44.748441 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:37:44.748781 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:37:44.749250 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:37:44.749263 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:37:44.750048 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:37:44.750051 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:37:44.750907 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:37:44.750915 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:37:44.751433 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:37:44.751437 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:37:44.751609 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:37:44.752241 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:37:44.752258 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:37:44.754250 (+ 1992us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:37:44.756193 (+ 1943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:37:44.756198 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:37:44.756210 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:37:44.756489 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:37:44.756768 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:37:44.756771 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:37:44.756773 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:37:44.756918 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:37:44.757302 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:37:44.757308 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:37:44.757551 (+ 243us) client_negotiation.cc:769] Sending connection context
0925 22:37:44.757729 (+ 178us) client_negotiation.cc:240] Negotiation successful
0925 22:37:44.758043 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":126,"threads_started":1}
I20260925 22:37:44.760023 18858 heartbeater.cc:347] Connected to a master server at 127.6.246.254:40061
I20260925 22:37:44.760250 18858 heartbeater.cc:464] Registering TS with master...
I20260925 22:37:44.760900 18858 heartbeater.cc:511] Master 127.6.246.254:40061 requested a full tablet report, sending...
I20260925 22:37:44.761947 18374 ts_manager.cc:194] Registered new tserver with Master: e1d8afb6c0d1493f83eb7f1ccfdcb062 (127.6.246.195:34385)
I20260925 22:37:44.762549 18374 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.246.195@KRBTEST.COM'} at 127.6.246.195:48357
I20260925 22:37:44.767216 7131 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:37:44.777199 18591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:37:44.769460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59592 (local address 127.6.246.254:40061)
0925 22:37:44.769604 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 22:37:44.769608 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:37:44.769837 (+ 229us) server_negotiation.cc:407] Connection header received
0925 22:37:44.769992 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:37:44.769995 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:37:44.770039 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:37:44.770125 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:37:44.770809 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:37:44.771322 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:37:44.772169 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:37:44.772406 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:37:44.772535 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:37:44.773998 (+ 1463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:37:44.774003 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:37:44.774008 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:37:44.774035 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:37:44.775501 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:37:44.775928 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:37:44.775935 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:37:44.775936 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:37:44.776015 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:37:44.776284 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:37:44.776289 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:37:44.776291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:37:44.776520 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:37:44.776659 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 22:37:44.776963 (+ 304us) server_negotiation.cc:299] Negotiation successful
0925 22:37:44.777082 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 22:37:44.780519 18374 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59592:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:37:44.783640 18374 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:37:44.800390 18873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:37:44.793170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.194:41483 (local address 127.0.0.1:57712)
0925 22:37:44.794371 (+ 1201us) negotiation.cc:107] Waiting for socket to connect
0925 22:37:44.794385 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:37:44.794510 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:37:44.795428 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:37:44.795435 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:37:44.795457 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:37:44.795581 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:37:44.795587 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:37:44.797097 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:37:44.797102 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:37:44.798505 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:37:44.798515 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:37:44.799815 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:37:44.799819 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:37:44.799950 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:37:44.799990 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 22:37:44.800092 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 22:37:44.800216 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1127,"thread_start_us":83,"threads_started":1}
I20260925 22:37:44.801249 18871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:37:44.792502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.195:34385 (local address 127.0.0.1:38476)
0925 22:37:44.793950 (+ 1448us) negotiation.cc:107] Waiting for socket to connect
0925 22:37:44.794008 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 22:37:44.794135 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:37:44.795573 (+ 1438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:37:44.795575 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:37:44.795589 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:37:44.795680 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:37:44.795687 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:37:44.797781 (+ 2094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:37:44.797785 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:37:44.798951 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:37:44.798960 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:37:44.800812 (+ 1852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:37:44.800816 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:37:44.800913 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:37:44.800941 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 22:37:44.801004 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 22:37:44.801080 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1167,"thread_start_us":101,"threads_started":1}
I20260925 22:37:44.801776 18875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:37:44.793895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38476 (local address 127.6.246.195:34385)
0925 22:37:44.795028 (+ 1133us) server_negotiation.cc:206] Beginning negotiation
0925 22:37:44.795039 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 22:37:44.795059 (+ 20us) server_negotiation.cc:407] Connection header received
0925 22:37:44.795128 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:37:44.795133 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:37:44.795287 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:37:44.795444 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:37:44.795798 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:37:44.797656 (+ 1858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:37:44.799102 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:37:44.799875 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:37:44.799971 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:37:44.801462 (+ 1491us) server_negotiation.cc:1035] Waiting for connection context
0925 22:37:44.801531 (+ 69us) server_negotiation.cc:299] Negotiation successful
0925 22:37:44.801632 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1039,"thread_start_us":128,"threads_started":1}
I20260925 22:37:44.802263 18872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:37:44.793328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57712 (local address 127.6.246.194:41483)
0925 22:37:44.794753 (+ 1425us) server_negotiation.cc:206] Beginning negotiation
0925 22:37:44.794767 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 22:37:44.794790 (+ 23us) server_negotiation.cc:407] Connection header received
0925 22:37:44.794868 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:37:44.794889 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:37:44.795105 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:37:44.795248 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:37:44.795710 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:37:44.796956 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:37:44.798683 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:37:44.799506 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:37:44.799609 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:37:44.799749 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 22:37:44.802027 (+ 2278us) server_negotiation.cc:299] Negotiation successful
0925 22:37:44.802128 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1315,"thread_start_us":123,"threads_started":1}
I20260925 22:37:44.804962 18792 tablet_service.cc:1467] Processing CreateTablet for tablet 5fa620f70f554c59bb852b9f09e03b3f (DEFAULT_TABLE table=test-table [id=e3a11b94132a4d758d8b23b28690f048]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:37:44.805568 18657 tablet_service.cc:1467] Processing CreateTablet for tablet 5fa620f70f554c59bb852b9f09e03b3f (DEFAULT_TABLE table=test-table [id=e3a11b94132a4d758d8b23b28690f048]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:37:44.806098 18792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fa620f70f554c59bb852b9f09e03b3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:37:44.806511 18657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fa620f70f554c59bb852b9f09e03b3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:37:44.811167 18873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:37:44.793481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.193:43189 (local address 127.0.0.1:59704)
0925 22:37:44.801023 (+ 7542us) negotiation.cc:107] Waiting for socket to connect
0925 22:37:44.801036 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:37:44.801093 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:37:44.802864 (+ 1771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:37:44.802867 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:37:44.802895 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:37:44.802963 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:37:44.802968 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:37:44.809052 (+ 6084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:37:44.809059 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:37:44.809779 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:37:44.809787 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:37:44.810775 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:37:44.810779 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:37:44.810865 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:37:44.810889 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 22:37:44.810953 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 22:37:44.811032 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7479,"thread_start_us":71,"threads_started":1}
I20260925 22:37:44.811487 18874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:37:44.793664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59704 (local address 127.6.246.193:43189)
0925 22:37:44.801637 (+ 7973us) server_negotiation.cc:206] Beginning negotiation
0925 22:37:44.801644 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:37:44.801668 (+ 24us) server_negotiation.cc:407] Connection header received
0925 22:37:44.801740 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:37:44.801745 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:37:44.802579 (+ 834us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:37:44.802725 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:37:44.807894 (+ 5169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:37:44.808930 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:37:44.809918 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:37:44.810645 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:37:44.810762 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:37:44.810961 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0925 22:37:44.811010 (+ 49us) server_negotiation.cc:299] Negotiation successful
0925 22:37:44.811095 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7560,"thread_start_us":93,"threads_started":1}
I20260925 22:37:44.814010 18515 tablet_service.cc:1467] Processing CreateTablet for tablet 5fa620f70f554c59bb852b9f09e03b3f (DEFAULT_TABLE table=test-table [id=e3a11b94132a4d758d8b23b28690f048]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:37:44.814602 18876 tablet_bootstrap.cc:492] T 5fa620f70f554c59bb852b9f09e03b3f P e1d8afb6c0d1493f83eb7f1ccfdcb062: Bootstrap starting.
I20260925 22:37:44.814976 18515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fa620f70f554c59bb852b9f09e03b3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:37:44.819099 18876 tablet_bootstrap.cc:654] T 5fa620f70f554c59bb852b9f09e03b3f P e1d8afb6c0d1493f83eb7f1ccfdcb062: Neither blocks nor log segments found. Creating new log.
I20260925 22:37:44.819967 18876 log.cc:824] T 5fa620f70f554c59bb852b9f09e03b3f P e1d8afb6c0d1493f83eb7f1ccfdcb062: Log is configured to *not* fsync() on all Append() calls
I20260925 22:37:44.822161 18876 tablet_bootstrap.cc:492] T 5fa620f70f554c59bb852b9f09e03b3f P e1d8afb6c0d1493f83eb7f1ccfdcb062: No bootstrap required, opened a new log
I20260925 22:37:44.822358 18876 ts_tablet_manager.cc:1397] T 5fa620f70f554c59bb852b9f09e03b3f P e1d8afb6c0d1493f83eb7f1ccfdcb062: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260925 22:37:44.826478 18876 raft_consensus.cc:348] T 5fa620f70f554c59bb852b9f09e03b3f P e1d8afb6c0d1493f83eb7f1ccfdcb062 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1d8afb6c0d1493f83eb7f1ccfdcb062" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 34385 } } peers { permanent_uuid: "884a72306b8b4334b0b1f4e2f491be15" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 41483 } } peers { permanent_uuid: "d3a2f3bb98894f778343ad0985002b87" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 43189 } }
I20260925 22:37:44.826718 18876 raft_consensus.cc:374] T 5fa620f70f554c59bb852b9f09e03b3f P e1d8afb6c0d1493f83eb7f1ccfdcb062 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:37:44.826776 18876 raft_consensus.cc:729] T 5fa620f70f554c59bb852b9f09e03b3f P e1d8afb6c0d1493f83eb7f1ccfdcb062 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1d8afb6c0d1493f83eb7f1ccfdcb062, State: Initialized, Role: FOLLOWER
I20260925 22:37:44.827206 18876 consensus_queue.cc:260] T 5fa620f70f554c59bb852b9f09e03b3f P e1d8afb6c0d1493f83eb7f1ccfdcb062 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1d8afb6c0d1493f83eb7f1ccfdcb062" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 34385 } } peers { permanent_uuid: "884a72306b8b4334b0b1f4e2f491be15" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 41483 } } peers { permanent_uuid: "d3a2f3bb98894f778343ad0985002b87" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 43189 } }
I20260925 22:37:44.828048 18876 ts_tablet_manager.cc:1428] T 5fa620f70f554c59bb852b9f09e03b3f P e1d8afb6c0d1493f83eb7f1ccfdcb062: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260925 22:37:44.828418 18858 heartbeater.cc:503] Master 127.6.246.254:40061 was elected leader, sending a full tablet report...
I20260925 22:37:44.828814 18879 tablet_bootstrap.cc:492] T 5fa620f70f554c59bb852b9f09e03b3f P d3a2f3bb98894f778343ad0985002b87: Bootstrap starting.
I20260925 22:37:44.828822 18878 tablet_bootstrap.cc:492] T 5fa620f70f554c59bb852b9f09e03b3f P 884a72306b8b4334b0b1f4e2f491be15: Bootstrap starting.
I20260925 22:37:44.831197 18879 tablet_bootstrap.cc:654] T 5fa620f70f554c59bb852b9f09e03b3f P d3a2f3bb98894f778343ad0985002b87: Neither blocks nor log segments found. Creating new log.
I20260925 22:37:44.831490 18878 tablet_bootstrap.cc:654] T 5fa620f70f554c59bb852b9f09e03b3f P 884a72306b8b4334b0b1f4e2f491be15: Neither blocks nor log segments found. Creating new log.
I20260925 22:37:44.832429 18879 log.cc:824] T 5fa620f70f554c59bb852b9f09e03b3f P d3a2f3bb98894f778343ad0985002b87: Log is configured to *not* fsync() on all Append() calls
I20260925 22:37:44.832439 18878 log.cc:824] T 5fa620f70f554c59bb852b9f09e03b3f P 884a72306b8b4334b0b1f4e2f491be15: Log is configured to *not* fsync() on all Append() calls
I20260925 22:37:44.834583 18879 tablet_bootstrap.cc:492] T 5fa620f70f554c59bb852b9f09e03b3f P d3a2f3bb98894f778343ad0985002b87: No bootstrap required, opened a new log
I20260925 22:37:44.834793 18879 ts_tablet_manager.cc:1397] T 5fa620f70f554c59bb852b9f09e03b3f P d3a2f3bb98894f778343ad0985002b87: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260925 22:37:44.838289 18879 raft_consensus.cc:348] T 5fa620f70f554c59bb852b9f09e03b3f P d3a2f3bb98894f778343ad0985002b87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1d8afb6c0d1493f83eb7f1ccfdcb062" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 34385 } } peers { permanent_uuid: "884a72306b8b4334b0b1f4e2f491be15" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 41483 } } peers { permanent_uuid: "d3a2f3bb98894f778343ad0985002b87" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 43189 } }
I20260925 22:37:44.838572 18879 raft_consensus.cc:374] T 5fa620f70f554c59bb852b9f09e03b3f P d3a2f3bb98894f778343ad0985002b87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:37:44.838640 18879 raft_consensus.cc:729] T 5fa620f70f554c59bb852b9f09e03b3f P d3a2f3bb98894f778343ad0985002b87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a2f3bb98894f778343ad0985002b87, State: Initialized, Role: FOLLOWER
I20260925 22:37:44.839100 18879 consensus_queue.cc:260] T 5fa620f70f554c59bb852b9f09e03b3f P d3a2f3bb98894f778343ad0985002b87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1d8afb6c0d1493f83eb7f1ccfdcb062" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 34385 } } peers { permanent_uuid: "884a72306b8b4334b0b1f4e2f491be15" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 41483 } } peers { permanent_uuid: "d3a2f3bb98894f778343ad0985002b87" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 43189 } }
I20260925 22:37:44.839761 18588 heartbeater.cc:503] Master 127.6.246.254:40061 was elected leader, sending a full tablet report...
I20260925 22:37:44.840188 18879 ts_tablet_manager.cc:1428] T 5fa620f70f554c59bb852b9f09e03b3f P d3a2f3bb98894f778343ad0985002b87: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260925 22:37:44.841176 18878 tablet_bootstrap.cc:492] T 5fa620f70f554c59bb852b9f09e03b3f P 884a72306b8b4334b0b1f4e2f491be15: No bootstrap required, opened a new log
I20260925 22:37:44.841370 18878 ts_tablet_manager.cc:1397] T 5fa620f70f554c59bb852b9f09e03b3f P 884a72306b8b4334b0b1f4e2f491be15: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.005s
I20260925 22:37:44.845683 18878 raft_consensus.cc:348] T 5fa620f70f554c59bb852b9f09e03b3f P 884a72306b8b4334b0b1f4e2f491be15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1d8afb6c0d1493f83eb7f1ccfdcb062" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 34385 } } peers { permanent_uuid: "884a72306b8b4334b0b1f4e2f491be15" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 41483 } } peers { permanent_uuid: "d3a2f3bb98894f778343ad0985002b87" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 43189 } }
I20260925 22:37:44.846021 18878 raft_consensus.cc:374] T 5fa620f70f554c59bb852b9f09e03b3f P 884a72306b8b4334b0b1f4e2f491be15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:37:44.846081 18878 raft_consensus.cc:729] T 5fa620f70f554c59bb852b9f09e03b3f P 884a72306b8b4334b0b1f4e2f491be15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 884a72306b8b4334b0b1f4e2f491be15, State: Initialized, Role: FOLLOWER
I20260925 22:37:44.846508 18878 consensus_queue.cc:260] T 5fa620f70f554c59bb852b9f09e03b3f P 884a72306b8b4334b0b1f4e2f491be15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1d8afb6c0d1493f83eb7f1ccfdcb062" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 34385 } } peers { permanent_uuid: "884a72306b8b4334b0b1f4e2f491be15" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 41483 } } peers { permanent_uuid: "d3a2f3bb98894f778343ad0985002b87" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 43189 } }
W20260925 22:37:44.846809 18589 tablet.cc:2369] T 5fa620f70f554c59bb852b9f09e03b3f P d3a2f3bb98894f778343ad0985002b87: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:37:44.847350 18724 heartbeater.cc:503] Master 127.6.246.254:40061 was elected leader, sending a full tablet report...
I20260925 22:37:44.847460 18878 ts_tablet_manager.cc:1428] T 5fa620f70f554c59bb852b9f09e03b3f P 884a72306b8b4334b0b1f4e2f491be15: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
W20260925 22:37:44.912002 18725 tablet.cc:2369] T 5fa620f70f554c59bb852b9f09e03b3f P 884a72306b8b4334b0b1f4e2f491be15: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 22:37:44 dist-test-slave-wjdr krb5kdc[17898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375864, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.254@KRBTEST.COM for HTTP/127.6.246.212@KRBTEST.COM
W20260925 22:37:44.994791 18859 tablet.cc:2369] T 5fa620f70f554c59bb852b9f09e03b3f P e1d8afb6c0d1493f83eb7f1ccfdcb062: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:37:45.083033 18883 raft_consensus.cc:482] T 5fa620f70f554c59bb852b9f09e03b3f P d3a2f3bb98894f778343ad0985002b87 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:37:45.083299 18883 raft_consensus.cc:504] T 5fa620f70f554c59bb852b9f09e03b3f P d3a2f3bb98894f778343ad0985002b87 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1d8afb6c0d1493f83eb7f1ccfdcb062" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 34385 } } peers { permanent_uuid: "884a72306b8b4334b0b1f4e2f491be15" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 41483 } } peers { permanent_uuid: "d3a2f3bb98894f778343ad0985002b87" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 43189 } }
I20260925 22:37:45.084777 18883 leader_election.cc:288] T 5fa620f70f554c59bb852b9f09e03b3f P d3a2f3bb98894f778343ad0985002b87 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1d8afb6c0d1493f83eb7f1ccfdcb062 (127.6.246.195:34385), 884a72306b8b4334b0b1f4e2f491be15 (127.6.246.194:41483)
I20260925 22:37:45.092227 18875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:37:45.085823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.193:39223 (local address 127.6.246.195:34385)
0925 22:37:45.085968 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 22:37:45.085973 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:37:45.087472 (+ 1499us) server_negotiation.cc:407] Connection header received
0925 22:37:45.087690 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:37:45.087694 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:37:45.087769 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:37:45.087824 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:37:45.088306 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:37:45.089130 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:37:45.090883 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:37:45.091489 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:37:45.091588 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:37:45.091641 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 22:37:45.092008 (+ 367us) server_negotiation.cc:299] Negotiation successful
0925 22:37:45.092089 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 22:37:45.092724 18887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:37:45.085057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.195:34385 (local address 127.6.246.193:39223)
0925 22:37:45.087362 (+ 2305us) negotiation.cc:107] Waiting for socket to connect
0925 22:37:45.087378 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:37:45.087556 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:37:45.087985 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:37:45.087989 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:37:45.088022 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:37:45.088168 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:37:45.088175 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:37:45.089301 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:37:45.089304 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:37:45.090735 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:37:45.090747 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:37:45.091754 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:37:45.091757 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:37:45.091854 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:37:45.091879 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 22:37:45.092479 (+ 600us) client_negotiation.cc:240] Negotiation successful
0925 22:37:45.092570 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2229,"thread_start_us":115,"threads_started":1}
I20260925 22:37:45.093469 18812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fa620f70f554c59bb852b9f09e03b3f" candidate_uuid: "d3a2f3bb98894f778343ad0985002b87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1d8afb6c0d1493f83eb7f1ccfdcb062" is_pre_election: true
I20260925 22:37:45.093865 18812 raft_consensus.cc:2454] T 5fa620f70f554c59bb852b9f09e03b3f P e1d8afb6c0d1493f83eb7f1ccfdcb062 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a2f3bb98894f778343ad0985002b87 in term 0.
I20260925 22:37:45.097325 18872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:37:45.085883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.193:45653 (local address 127.6.246.194:41483)
0925 22:37:45.086005 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 22:37:45.086009 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:37:45.092819 (+ 6810us) server_negotiation.cc:407] Connection header received
0925 22:37:45.093140 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:37:45.093143 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:37:45.093198 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:37:45.093246 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:37:45.093652 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:37:45.094407 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:37:45.095995 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:37:45.096612 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:37:45.096706 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:37:45.096755 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 22:37:45.097123 (+ 368us) server_negotiation.cc:299] Negotiation successful
0925 22:37:45.097205 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 22:37:45.097898 18888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:37:45.085471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.194:41483 (local address 127.6.246.193:45653)
0925 22:37:45.092713 (+ 7242us) negotiation.cc:107] Waiting for socket to connect
0925 22:37:45.092729 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:37:45.093002 (+ 273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:37:45.093377 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:37:45.093380 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:37:45.093396 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:37:45.093503 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:37:45.093509 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:37:45.094618 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:37:45.094621 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:37:45.095808 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:37:45.095823 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:37:45.096869 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:37:45.096873 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:37:45.096974 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:37:45.096998 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 22:37:45.097669 (+ 671us) client_negotiation.cc:240] Negotiation successful
0925 22:37:45.097744 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7168,"thread_start_us":95,"threads_started":1}
I20260925 22:37:45.098198 18475 leader_election.cc:302] T 5fa620f70f554c59bb852b9f09e03b3f P d3a2f3bb98894f778343ad0985002b87 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a2f3bb98894f778343ad0985002b87, e1d8afb6c0d1493f83eb7f1ccfdcb062; no voters:
I20260925 22:37:45.098484 18883 raft_consensus.cc:2781] T 5fa620f70f554c59bb852b9f09e03b3f P d3a2f3bb98894f778343ad0985002b87 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:37:45.098570 18883 raft_consensus.cc:482] T 5fa620f70f554c59bb852b9f09e03b3f P d3a2f3bb98894f778343ad0985002b87 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:37:45.098609 18883 raft_consensus.cc:3037] T 5fa620f70f554c59bb852b9f09e03b3f P d3a2f3bb98894f778343ad0985002b87 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:37:45.099189 18662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fa620f70f554c59bb852b9f09e03b3f" candidate_uuid: "d3a2f3bb98894f778343ad0985002b87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "884a72306b8b4334b0b1f4e2f491be15" is_pre_election: true
I20260925 22:37:45.099485 18662 raft_consensus.cc:2454] T 5fa620f70f554c59bb852b9f09e03b3f P 884a72306b8b4334b0b1f4e2f491be15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a2f3bb98894f778343ad0985002b87 in term 0.
I20260925 22:37:45.099924 18883 raft_consensus.cc:504] T 5fa620f70f554c59bb852b9f09e03b3f P d3a2f3bb98894f778343ad0985002b87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1d8afb6c0d1493f83eb7f1ccfdcb062" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 34385 } } peers { permanent_uuid: "884a72306b8b4334b0b1f4e2f491be15" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 41483 } } peers { permanent_uuid: "d3a2f3bb98894f778343ad0985002b87" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 43189 } }
I20260925 22:37:45.100283 18883 leader_election.cc:288] T 5fa620f70f554c59bb852b9f09e03b3f P d3a2f3bb98894f778343ad0985002b87 [CANDIDATE]: Term 1 election: Requested vote from peers e1d8afb6c0d1493f83eb7f1ccfdcb062 (127.6.246.195:34385), 884a72306b8b4334b0b1f4e2f491be15 (127.6.246.194:41483)
I20260925 22:37:45.100924 18662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fa620f70f554c59bb852b9f09e03b3f" candidate_uuid: "d3a2f3bb98894f778343ad0985002b87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "884a72306b8b4334b0b1f4e2f491be15"
I20260925 22:37:45.100950 18812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fa620f70f554c59bb852b9f09e03b3f" candidate_uuid: "d3a2f3bb98894f778343ad0985002b87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1d8afb6c0d1493f83eb7f1ccfdcb062"
I20260925 22:37:45.101042 18662 raft_consensus.cc:3037] T 5fa620f70f554c59bb852b9f09e03b3f P 884a72306b8b4334b0b1f4e2f491be15 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:37:45.101084 18812 raft_consensus.cc:3037] T 5fa620f70f554c59bb852b9f09e03b3f P e1d8afb6c0d1493f83eb7f1ccfdcb062 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:37:45.102264 18662 raft_consensus.cc:2454] T 5fa620f70f554c59bb852b9f09e03b3f P 884a72306b8b4334b0b1f4e2f491be15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a2f3bb98894f778343ad0985002b87 in term 1.
I20260925 22:37:45.102452 18812 raft_consensus.cc:2454] T 5fa620f70f554c59bb852b9f09e03b3f P e1d8afb6c0d1493f83eb7f1ccfdcb062 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a2f3bb98894f778343ad0985002b87 in term 1.
I20260925 22:37:45.102849 18475 leader_election.cc:302] T 5fa620f70f554c59bb852b9f09e03b3f P d3a2f3bb98894f778343ad0985002b87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 884a72306b8b4334b0b1f4e2f491be15, d3a2f3bb98894f778343ad0985002b87; no voters:
I20260925 22:37:45.103169 18883 raft_consensus.cc:2781] T 5fa620f70f554c59bb852b9f09e03b3f P d3a2f3bb98894f778343ad0985002b87 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:37:45.103382 18883 raft_consensus.cc:686] T 5fa620f70f554c59bb852b9f09e03b3f P d3a2f3bb98894f778343ad0985002b87 [term 1 LEADER]: Becoming Leader. State: Replica: d3a2f3bb98894f778343ad0985002b87, State: Running, Role: LEADER
I20260925 22:37:45.103719 18883 consensus_queue.cc:237] T 5fa620f70f554c59bb852b9f09e03b3f P d3a2f3bb98894f778343ad0985002b87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1d8afb6c0d1493f83eb7f1ccfdcb062" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 34385 } } peers { permanent_uuid: "884a72306b8b4334b0b1f4e2f491be15" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 41483 } } peers { permanent_uuid: "d3a2f3bb98894f778343ad0985002b87" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 43189 } }
I20260925 22:37:45.107393 18374 catalog_manager.cc:5495] T 5fa620f70f554c59bb852b9f09e03b3f P d3a2f3bb98894f778343ad0985002b87 reported cstate change: term changed from 0 to 1, leader changed from <none> to d3a2f3bb98894f778343ad0985002b87 (127.6.246.193). New cstate: current_term: 1 leader_uuid: "d3a2f3bb98894f778343ad0985002b87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1d8afb6c0d1493f83eb7f1ccfdcb062" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 34385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "884a72306b8b4334b0b1f4e2f491be15" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 41483 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a2f3bb98894f778343ad0985002b87" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 43189 } health_report { overall_health: HEALTHY } } }
I20260925 22:37:45.167879 18874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:37:45.162822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59710 (local address 127.6.246.193:43189)
0925 22:37:45.162939 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0925 22:37:45.162944 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:37:45.162958 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:37:45.163047 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:37:45.163050 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:37:45.163126 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:37:45.163239 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:37:45.163636 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:37:45.164430 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:37:45.165711 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:37:45.166148 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:37:45.166271 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:37:45.166683 (+ 412us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:37:45.167305 (+ 622us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:37:45.167463 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 22:37:45.167646 (+ 183us) server_negotiation.cc:299] Negotiation successful
0925 22:37:45.167705 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 22:37:45.174314 18662 raft_consensus.cc:1261] T 5fa620f70f554c59bb852b9f09e03b3f P 884a72306b8b4334b0b1f4e2f491be15 [term 1 FOLLOWER]: Refusing update from remote peer d3a2f3bb98894f778343ad0985002b87: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:37:45.174383 18812 raft_consensus.cc:1261] T 5fa620f70f554c59bb852b9f09e03b3f P e1d8afb6c0d1493f83eb7f1ccfdcb062 [term 1 FOLLOWER]: Refusing update from remote peer d3a2f3bb98894f778343ad0985002b87: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:37:45.175031 18883 consensus_queue.cc:1035] T 5fa620f70f554c59bb852b9f09e03b3f P d3a2f3bb98894f778343ad0985002b87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "884a72306b8b4334b0b1f4e2f491be15" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 41483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:37:45.175272 18889 consensus_queue.cc:1035] T 5fa620f70f554c59bb852b9f09e03b3f P d3a2f3bb98894f778343ad0985002b87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1d8afb6c0d1493f83eb7f1ccfdcb062" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 34385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:37:45.182353 18899 mvcc.cc:205] Tried to move back new op lower bound from 7333379543746580480 to 7333379543468769280. Current Snapshot: MvccSnapshot[applied={T|T < 7333379543746580480}]
I20260925 22:37:45.184567 18900 mvcc.cc:205] Tried to move back new op lower bound from 7333379543746580480 to 7333379543468769280. Current Snapshot: MvccSnapshot[applied={T|T < 7333379543746580480}]
I20260925 22:37:45.196717 18373 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:37:45.196902 18373 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:37:45.199478 18373 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8608c8d900de46cdb8fe5ae31c2e9d4b: Sending DeleteTablet for 3 replicas of tablet 5fa620f70f554c59bb852b9f09e03b3f
I20260925 22:37:45.200278 18657 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fa620f70f554c59bb852b9f09e03b3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:37:45 UTC) from {username='kudu', principal='kudu/127.6.246.254@KRBTEST.COM'} at 127.0.0.1:57712
I20260925 22:37:45.200362 18515 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fa620f70f554c59bb852b9f09e03b3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:37:45 UTC) from {username='kudu', principal='kudu/127.6.246.254@KRBTEST.COM'} at 127.0.0.1:59704
I20260925 22:37:45.200798 18908 tablet_replica.cc:330] T 5fa620f70f554c59bb852b9f09e03b3f P 884a72306b8b4334b0b1f4e2f491be15: stopping tablet replica
I20260925 22:37:45.201112 18792 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fa620f70f554c59bb852b9f09e03b3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:37:45 UTC) from {username='kudu', principal='kudu/127.6.246.254@KRBTEST.COM'} at 127.0.0.1:38476
I20260925 22:37:45.201170 18908 raft_consensus.cc:2229] T 5fa620f70f554c59bb852b9f09e03b3f P 884a72306b8b4334b0b1f4e2f491be15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:37:45.201589 18908 raft_consensus.cc:2258] T 5fa620f70f554c59bb852b9f09e03b3f P 884a72306b8b4334b0b1f4e2f491be15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:37:45.201656 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 18455
I20260925 22:37:45.203900 18908 ts_tablet_manager.cc:1905] T 5fa620f70f554c59bb852b9f09e03b3f P 884a72306b8b4334b0b1f4e2f491be15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:37:45.206732 18908 ts_tablet_manager.cc:1918] T 5fa620f70f554c59bb852b9f09e03b3f P 884a72306b8b4334b0b1f4e2f491be15: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:37:45.206831 18908 log.cc:1196] T 5fa620f70f554c59bb852b9f09e03b3f P 884a72306b8b4334b0b1f4e2f491be15: Deleting WAL directory at /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/ts-1/wal/wals/5fa620f70f554c59bb852b9f09e03b3f
I20260925 22:37:45.207113 18910 tablet_replica.cc:330] T 5fa620f70f554c59bb852b9f09e03b3f P e1d8afb6c0d1493f83eb7f1ccfdcb062: stopping tablet replica
I20260925 22:37:45.207152 18908 ts_tablet_manager.cc:1939] T 5fa620f70f554c59bb852b9f09e03b3f P 884a72306b8b4334b0b1f4e2f491be15: Deleting consensus metadata
I20260925 22:37:45.207542 18910 raft_consensus.cc:2229] T 5fa620f70f554c59bb852b9f09e03b3f P e1d8afb6c0d1493f83eb7f1ccfdcb062 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:37:45.207906 18910 raft_consensus.cc:2258] T 5fa620f70f554c59bb852b9f09e03b3f P e1d8afb6c0d1493f83eb7f1ccfdcb062 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:37:45.208376 18358 catalog_manager.cc:4841] TS 884a72306b8b4334b0b1f4e2f491be15 (127.6.246.194:41483): tablet 5fa620f70f554c59bb852b9f09e03b3f (table test-table [id=e3a11b94132a4d758d8b23b28690f048]) successfully deleted
W20260925 22:37:45.208660 18358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.246.193:43189 (error 108)
I20260925 22:37:45.208787 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 18594
I20260925 22:37:45.210291 18873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:37:45.209983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.193:43189 (local address 127.0.0.1:59722)
0925 22:37:45.210142 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 22:37:45.210211 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.193:43189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260925 22:37:45.210476 18358 catalog_manager.cc:4584] TS d3a2f3bb98894f778343ad0985002b87 (127.6.246.193:43189): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5fa620f70f554c59bb852b9f09e03b3f: Network error: Client connection negotiation failed: client connection to 127.6.246.193:43189: connect: Connection refused (error 111)
I20260925 22:37:45.210517 18910 ts_tablet_manager.cc:1905] T 5fa620f70f554c59bb852b9f09e03b3f P e1d8afb6c0d1493f83eb7f1ccfdcb062: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:37:45.213095 18910 ts_tablet_manager.cc:1918] T 5fa620f70f554c59bb852b9f09e03b3f P e1d8afb6c0d1493f83eb7f1ccfdcb062: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:37:45.213201 18910 log.cc:1196] T 5fa620f70f554c59bb852b9f09e03b3f P e1d8afb6c0d1493f83eb7f1ccfdcb062: Deleting WAL directory at /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790375752797504-7131-0/minicluster-data/ts-2/wal/wals/5fa620f70f554c59bb852b9f09e03b3f
I20260925 22:37:45.213527 18910 ts_tablet_manager.cc:1939] T 5fa620f70f554c59bb852b9f09e03b3f P e1d8afb6c0d1493f83eb7f1ccfdcb062: Deleting consensus metadata
I20260925 22:37:45.214327 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 18729
I20260925 22:37:45.219897 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 18342
I20260925 22:37:45.231663 7131 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 22:37:45.566336 7131 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 22:37:45.566488 7131 mini_ranger.cc:66] Stopping Ranger...
I20260925 22:37:45.911497 7131 mini_ranger.cc:68] Stopped Ranger
2026-09-25 22:37:45.911 UTC [17928] LOG: received smart shutdown request
2026-09-25 22:37:45.915 UTC [17928] LOG: background worker "logical replication launcher" (PID 17937) exited with exit code 1
2026-09-25 22:37:45.915 UTC [17932] LOG: shutting down
2026-09-25 22:37:46.088 UTC [17928] LOG: database system is shut down
2026-09-25T22:37:46Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (47642 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:37:46 dist-test-slave-wjdr krb5kdc[18920](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:37:46 dist-test-slave-wjdr krb5kdc[18920](info): set up 2 sockets
Sep 25 22:37:46 dist-test-slave-wjdr krb5kdc[18920](info): commencing operation
krb5kdc: starting...
W20260925 22:37:47.284798 7131 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Sep 25 22:37:47 dist-test-slave-wjdr krb5kdc[18920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375867, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:37:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:37:47Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.6.246.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.6.246.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.6.246.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/krb5kdc/rangeradmin_127.6.246.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.6.246.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/krb5kdc/rangeradmin_127.6.246.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.6.246.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.6.246.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.6.246.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/krb5kdc/rangerlookup_127.6.246.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.6.246.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/krb5kdc/rangerlookup_127.6.246.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.246.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.246.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.246.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/krb5kdc/HTTP_127.6.246.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.246.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/krb5kdc/HTTP_127.6.246.212@KRBTEST.COM.keytab.
I20260925 22:37:47.456197 7131 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskan1x8t/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/postgres -l logfile start
2026-09-25 22:37:50.153 UTC [18950] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 22:37:50.153 UTC [18950] LOG: listening on IPv4 address "127.6.246.212", port 34993
2026-09-25 22:37:50.157 UTC [18950] LOG: listening on Unix socket "/tmp/.s.PGSQL.34993"
2026-09-25 22:37:50.181 UTC [18953] LOG: database system was shut down at 2026-09-25 22:37:48 UTC
2026-09-25 22:37:50.185 UTC [18950] LOG: database system is ready to accept connections
I20260925 22:37:51.136864 7131 mini_postgres.cc:96] Postgres bound to 34993
2026-09-25 22:37:51.144 UTC [18961] FATAL: database "slave" does not exist
127.6.246.212:34993 - accepting connections
I20260925 22:37:51.145406 7131 mini_ranger.cc:161] Starting Ranger...
I20260925 22:37:51.162443 7131 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 22:37:51.674330 7131 mini_ranger.cc:87] Created ranger Postgres database
I20260925 22:37:51.674446 7131 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ranger-admin
2026-09-25 22:37:51,954 [I] DB FLAVOR :POSTGRES
2026-09-25 22:37:51,954 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 22:37:51,955 [I] Checking connection..
2026-09-25 22:37:51,955 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskan1x8t/build/debug/bin/postgresql.jar:/tmp/dist-test-taskan1x8t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.246.212:34993/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 22:37:52,238 [I] Checking connection passed.
2026-09-25 22:37:52,238 [I] --------- Verifying version history table ---------
2026-09-25 22:37:52,238 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskan1x8t/build/debug/bin/postgresql.jar:/tmp/dist-test-taskan1x8t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.246.212:34993/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 22:37:52,482 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 22:37:52,482 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskan1x8t/build/debug/bin/postgresql.jar:/tmp/dist-test-taskan1x8t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.246.212:34993/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 22:37:52,708 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 22:37:52,709 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 22:37:52,709 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskan1x8t/build/debug/bin/postgresql.jar:/tmp/dist-test-taskan1x8t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.246.212:34993/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskan1x8t/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 22:37:52.958 UTC [19025] WARNING: there is no transaction in progress
2026-09-25 22:37:52,964 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 22:37:52,965 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskan1x8t/build/debug/bin/postgresql.jar:/tmp/dist-test-taskan1x8t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.246.212:34993/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 22:37:53,216 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 22:37:53,216 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 22:37:53,216 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskan1x8t/build/debug/bin/postgresql.jar:/tmp/dist-test-taskan1x8t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.246.212:34993/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 22:37:53,467 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskan1x8t/build/debug/bin/postgresql.jar:/tmp/dist-test-taskan1x8t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.246.212:34993/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 22:37:53,712 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskan1x8t/build/debug/bin/postgresql.jar:/tmp/dist-test-taskan1x8t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.246.212:34993/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-wjdr.c.gcp-upstream.internal','N') ;"
2026-09-25 22:37:53,958 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskan1x8t/build/debug/bin/postgresql.jar:/tmp/dist-test-taskan1x8t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.246.212:34993/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 22:37:54,197 [I] Table x_portal_user does not exist in database ranger
2026-09-25 22:37:54,197 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskan1x8t/build/debug/bin/postgresql.jar:/tmp/dist-test-taskan1x8t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.246.212:34993/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 22:37:54,445 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 22:37:54,446 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskan1x8t/build/debug/bin/postgresql.jar:/tmp/dist-test-taskan1x8t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.246.212:34993/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 22:37:54,697 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskan1x8t/build/debug/bin/postgresql.jar:/tmp/dist-test-taskan1x8t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.246.212:34993/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 22:37:54,927 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 22:37:54,927 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskan1x8t/build/debug/bin/postgresql.jar:/tmp/dist-test-taskan1x8t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.246.212:34993/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskan1x8t/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 22:37:55.954 UTC [19133] WARNING: there is no transaction in progress
2026-09-25 22:37:55.973 UTC [19133] WARNING: there is no transaction in progress
2026-09-25 22:37:55.991 UTC [19133] WARNING: there is no transaction in progress
2026-09-25 22:37:56.008 UTC [19133] WARNING: there is no transaction in progress
2026-09-25 22:37:56.029 UTC [19133] WARNING: there is no transaction in progress
2026-09-25 22:37:56.050 UTC [19133] WARNING: there is no transaction in progress
2026-09-25 22:37:56.109 UTC [19133] WARNING: there is no transaction in progress
2026-09-25 22:37:56.117 UTC [19133] WARNING: there is no transaction in progress
2026-09-25 22:37:56.132 UTC [19133] WARNING: there is no transaction in progress
2026-09-25 22:37:56.144 UTC [19133] WARNING: there is no transaction in progress
2026-09-25 22:37:56.154 UTC [19133] WARNING: there is no transaction in progress
2026-09-25 22:37:56.169 UTC [19133] WARNING: there is no transaction in progress
2026-09-25 22:37:56.179 UTC [19133] WARNING: there is no transaction in progress
2026-09-25 22:37:56.186 UTC [19133] WARNING: there is no transaction in progress
2026-09-25 22:37:56.193 UTC [19133] WARNING: there is no transaction in progress
2026-09-25 22:37:56.201 UTC [19133] WARNING: there is no transaction in progress
2026-09-25 22:37:56,880 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 22:37:56,880 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskan1x8t/build/debug/bin/postgresql.jar:/tmp/dist-test-taskan1x8t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.246.212:34993/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 22:37:57,154 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 22:37:57,155 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskan1x8t/build/debug/bin/postgresql.jar:/tmp/dist-test-taskan1x8t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.246.212:34993/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 22:37:57,431 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 22:37:57,432 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskan1x8t/build/debug/bin/postgresql.jar:/tmp/dist-test-taskan1x8t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.246.212:34993/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 22:37:57,682 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 22:37:57,683 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskan1x8t/build/debug/bin/postgresql.jar:/tmp/dist-test-taskan1x8t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.246.212:34993/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 22:37:57,946 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskan1x8t/build/debug/bin/postgresql.jar:/tmp/dist-test-taskan1x8t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.246.212:34993/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 22:37:58,197 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskan1x8t/build/debug/bin/postgresql.jar:/tmp/dist-test-taskan1x8t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.246.212:34993/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-wjdr.c.gcp-upstream.internal';"
2026-09-25 22:37:58,432 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 22:37:58,432 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskan1x8t/build/debug/bin/postgresql.jar:/tmp/dist-test-taskan1x8t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.246.212:34993/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 22:37:58,660 [I] DB_PATCHES have already been applied
I20260925 22:37:58.666311 7131 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ranger-admin:/tmp/dist-test-taskan1x8t/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskan1x8t/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskan1x8t/build/debug/bin/postgresql.jar:/tmp/dist-test-taskan1x8t/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 22:37:58.666388 7131 mini_ranger.cc:193] Using host: 127.6.246.212
I20260925 22:37:58.669989 7131 mini_ranger.cc:227] Ranger admin URL: http://127.6.246.212:57669
Sep 25, 2026 10:37:58 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 10:37:58 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 10:37:59 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 10:37:59 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 10:37:59 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 10:37:59 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 10:37:59 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 22:37:59.776245 7131 mini_ranger.cc:160] Time spent starting Ranger: real 8.631s user 0.000s sys 0.011s
Sep 25, 2026 10:38:20 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 10:38:20 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 25, 2026 10:38:20 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 10:38:20 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 10:38:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 10:38:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 10:38:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 10:38:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 10:38:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 10:38:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 10:38:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 10:38:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 10:38:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 10:38:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 10:38:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 10:38:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 10:38:20 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 10:38:21 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 22:38:21.761771 7131 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.6.246.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.6.246.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.6.246.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/krb5kdc/rangerkms_127.6.246.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.6.246.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/krb5kdc/rangerkms_127.6.246.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.246.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.6.246.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.246.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/krb5kdc/HTTP_127.6.246.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.246.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/krb5kdc/HTTP_127.6.246.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 22:38:21 dist-test-slave-wjdr krb5kdc[18920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375901, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 22:38:21.852038 7131 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 22:38:21.862092 7131 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 22:38:22.689246 7131 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 22:38:22.689327 7131 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ranger-kms
I20260925 22:38:22.689342 7131 mini_ranger_kms.cc:227] Using postgres at 127.6.246.212:34993
2026-09-25 22:38:22,873 [I] DB FLAVOR :POSTGRES
2026-09-25 22:38:22,873 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 22:38:22,873 [I] Checking connection
2026-09-25 22:38:22,873 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskan1x8t/build/debug/bin/postgresql.jar:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.246.212:34993/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 22:38:23,127 [I] connection success
2026-09-25 22:38:23,127 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 22:38:23,128 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 22:38:23,128 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskan1x8t/build/debug/bin/postgresql.jar:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.246.212:34993/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 22:38:23,379 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 22:38:23,379 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 22:38:23,380 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 22:38:23,380 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskan1x8t/build/debug/bin/postgresql.jar:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.246.212:34993/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 22:38:23,660 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 22:38:24.116441 7131 mini_ranger_kms.cc:310] Created kms service
I20260925 22:38:24.307216 7131 mini_ranger_kms.cc:326] Created kudu user
I20260925 22:38:24.355140 7131 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 22:38:24 dist-test-slave-wjdr krb5kdc[18920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375901, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.6.246.212@KRBTEST.COM
I20260925 22:38:24.731801 7131 mini_ranger_kms.cc:384] Added ranger policy
I20260925 22:38:24.731963 7131 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ranger-kms:/tmp/dist-test-taskan1x8t/build/debug/bin/postgresql.jar:/tmp/dist-test-taskan1x8t/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskan1x8t/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 22:38:24.731986 7131 mini_ranger_kms.cc:240] Using host: 127.6.246.212
I20260925 22:38:24.736701 7131 mini_ranger_kms.cc:276] Ranger KMS PID: 19313
I20260925 22:38:24.736786 7131 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.6.246.212:39789
Sep 25, 2026 10:38:25 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 10:38:25 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 10:38:26 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 10:38:26 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 10:38:26 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 10:38:26 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260925 22:38:26.972265 7131 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.120s user 0.012s sys 0.155s
I20260925 22:38:26.972414 7131 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 22:38:26.972537 7131 mini_ranger_kms.cc:401] 127.6.246.212:39789/kms/v1/keys
Sep 25 22:38:27 dist-test-slave-wjdr krb5kdc[18920](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790375907, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.246.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 25 22:38:27 dist-test-slave-wjdr krb5kdc[18920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375907, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.246.212@KRBTEST.COM for HTTP/127.6.246.212@KRBTEST.COM
Sep 25 22:38:28 dist-test-slave-wjdr krb5kdc[18920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375907, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.246.212@KRBTEST.COM for HTTP/127.6.246.212@KRBTEST.COM
Sep 25 22:38:30 dist-test-slave-wjdr krb5kdc[18920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.6.246.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab.
Entry for principal kudu/127.6.246.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab.
I20260925 22:38:30.504505 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:41441
--webserver_interface=127.6.246.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab
--principal=kudu/127.6.246.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:43373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.246.254:41441
--ranger_config_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.246.212:39789/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:38:30.634862 19364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:38:30.635195 19364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:38:30.635264 19364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:38:30.639384 19364 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:38:30.639477 19364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:38:30.639514 19364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:38:30.639575 19364 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:38:30.639611 19364 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:38:30.645377 19364 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:43373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/master-0/wal
--ranger_kms_url=127.6.246.212:39789/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.6.246.254:41441
--ranger_config_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab
--principal=kudu/127.6.246.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:41441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.246.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19364
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:38:30.646541 19364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:38:30.647578 19364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:38:30.657723 19369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:38:30.657882 19364 server_base.cc:1034] running on GCE node
W20260925 22:38:30.657708 19372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:38:30.658221 19370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:38:30.658849 19364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:38:30.659979 19364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:38:30.661166 19364 hybrid_clock.cc:648] HybridClock initialized: now 1790375910661147 us; error 39 us; skew 500 ppm
Sep 25 22:38:30 dist-test-slave-wjdr krb5kdc[18920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375910, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:38:30.664196 19364 init.cc:377] Logged in from keytab as kudu/127.6.246.254@KRBTEST.COM (short username kudu)
I20260925 22:38:30.665261 19364 webserver.cc:460] Webserver started at http://127.6.246.254:33037/ using document root <none> and password file <none>
I20260925 22:38:30.665894 19364 fs_manager.cc:359] Metadata directory not provided
I20260925 22:38:30.665982 19364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:38:30.666206 19364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 22:38:30 dist-test-slave-wjdr krb5kdc[18920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375910, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.254@KRBTEST.COM for HTTP/127.6.246.212@KRBTEST.COM
I20260925 22:38:30.733559 19364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/master-0/data/instance:
uuid: "5ef6e669f7244e6e85a143e0e9fb5b20"
format_stamp: "Formatted at 2026-09-25 22:38:30 on dist-test-slave-wjdr"
server_key: "274c84d3b98c9b5640d7da684e9cbb92"
server_key_iv: "b8e72e075a28b1103712a966ebc06c40"
server_key_version: "kuduclusterkey@0"
I20260925 22:38:30.734277 19364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/master-0/wal/instance:
uuid: "5ef6e669f7244e6e85a143e0e9fb5b20"
format_stamp: "Formatted at 2026-09-25 22:38:30 on dist-test-slave-wjdr"
server_key: "274c84d3b98c9b5640d7da684e9cbb92"
server_key_iv: "b8e72e075a28b1103712a966ebc06c40"
server_key_version: "kuduclusterkey@0"
I20260925 22:38:30.738458 19364 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:38:30.753827 19379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:38:30.755267 19364 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:38:30.755431 19364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/master-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/master-0/wal
uuid: "5ef6e669f7244e6e85a143e0e9fb5b20"
format_stamp: "Formatted at 2026-09-25 22:38:30 on dist-test-slave-wjdr"
server_key: "274c84d3b98c9b5640d7da684e9cbb92"
server_key_iv: "b8e72e075a28b1103712a966ebc06c40"
server_key_version: "kuduclusterkey@0"
I20260925 22:38:30.755604 19364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:38:30.772418 19364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:38:30.775650 19364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:38:30.775880 19364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:38:30.784314 19364 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.254:41441
I20260925 22:38:30.784320 19431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.254:41441 every 8 connection(s)
I20260925 22:38:30.785390 19364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
I20260925 22:38:30.788398 19432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:38:30.793390 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 19364
I20260925 22:38:30.793522 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/master-0/wal/instance
I20260925 22:38:30.793840 7131 external_mini_cluster.cc:1428] Setting key 0d66aef993a6b17c6afdf04264b691b8
I20260925 22:38:30.794358 19432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ef6e669f7244e6e85a143e0e9fb5b20: Bootstrap starting.
I20260925 22:38:30.797263 19432 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ef6e669f7244e6e85a143e0e9fb5b20: Neither blocks nor log segments found. Creating new log.
I20260925 22:38:30.798159 19432 log.cc:824] T 00000000000000000000000000000000 P 5ef6e669f7244e6e85a143e0e9fb5b20: Log is configured to *not* fsync() on all Append() calls
I20260925 22:38:30.800427 19432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ef6e669f7244e6e85a143e0e9fb5b20: No bootstrap required, opened a new log
Sep 25 22:38:30 dist-test-slave-wjdr krb5kdc[18920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
I20260925 22:38:30.804759 19432 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ef6e669f7244e6e85a143e0e9fb5b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ef6e669f7244e6e85a143e0e9fb5b20" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 41441 } }
I20260925 22:38:30.805094 19432 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ef6e669f7244e6e85a143e0e9fb5b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:38:30.805168 19432 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ef6e669f7244e6e85a143e0e9fb5b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ef6e669f7244e6e85a143e0e9fb5b20, State: Initialized, Role: FOLLOWER
I20260925 22:38:30.805671 19432 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ef6e669f7244e6e85a143e0e9fb5b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ef6e669f7244e6e85a143e0e9fb5b20" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 41441 } }
I20260925 22:38:30.805863 19432 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ef6e669f7244e6e85a143e0e9fb5b20 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:38:30.805953 19432 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ef6e669f7244e6e85a143e0e9fb5b20 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:38:30.806082 19432 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ef6e669f7244e6e85a143e0e9fb5b20 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:38:30.806989 19432 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ef6e669f7244e6e85a143e0e9fb5b20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ef6e669f7244e6e85a143e0e9fb5b20" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 41441 } }
I20260925 22:38:30.807355 19432 leader_election.cc:302] T 00000000000000000000000000000000 P 5ef6e669f7244e6e85a143e0e9fb5b20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ef6e669f7244e6e85a143e0e9fb5b20; no voters:
I20260925 22:38:30.807745 19432 leader_election.cc:288] T 00000000000000000000000000000000 P 5ef6e669f7244e6e85a143e0e9fb5b20 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:38:30.807886 19437 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ef6e669f7244e6e85a143e0e9fb5b20 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:38:30.808136 19437 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ef6e669f7244e6e85a143e0e9fb5b20 [term 1 LEADER]: Becoming Leader. State: Replica: 5ef6e669f7244e6e85a143e0e9fb5b20, State: Running, Role: LEADER
I20260925 22:38:30.808478 19437 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ef6e669f7244e6e85a143e0e9fb5b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ef6e669f7244e6e85a143e0e9fb5b20" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 41441 } }
I20260925 22:38:30.809099 19432 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ef6e669f7244e6e85a143e0e9fb5b20 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:38:30.809617 19437 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ef6e669f7244e6e85a143e0e9fb5b20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ef6e669f7244e6e85a143e0e9fb5b20. Latest consensus state: current_term: 1 leader_uuid: "5ef6e669f7244e6e85a143e0e9fb5b20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ef6e669f7244e6e85a143e0e9fb5b20" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 41441 } } }
I20260925 22:38:30.809715 19437 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ef6e669f7244e6e85a143e0e9fb5b20 [sys.catalog]: This master's current role is: LEADER
I20260925 22:38:30.810032 19439 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ef6e669f7244e6e85a143e0e9fb5b20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ef6e669f7244e6e85a143e0e9fb5b20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ef6e669f7244e6e85a143e0e9fb5b20" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 41441 } } }
I20260925 22:38:30.810154 19439 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ef6e669f7244e6e85a143e0e9fb5b20 [sys.catalog]: This master's current role is: LEADER
I20260925 22:38:30.811295 19432 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 22:38:30.812219 19442 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:38:30.815639 19435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:30.795406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43222 (local address 127.6.246.254:41441)
0925 22:38:30.795888 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0925 22:38:30.795906 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 22:38:30.795948 (+ 42us) server_negotiation.cc:407] Connection header received
0925 22:38:30.796832 (+ 884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:38:30.796856 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:38:30.797283 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:38:30.797671 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:38:30.798525 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:30.799959 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:30.801157 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:30.801832 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:30.802022 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:30.805084 (+ 3062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:38:30.805091 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:38:30.805109 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:38:30.805140 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:38:30.807115 (+ 1975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:38:30.807625 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:38:30.807634 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:38:30.807642 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:38:30.807778 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:38:30.808130 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:38:30.808134 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:38:30.808136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:38:30.810764 (+ 2628us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:38:30.811756 (+ 992us) server_negotiation.cc:1035] Waiting for connection context
0925 22:38:30.811903 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 22:38:30.812164 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":324,"thread_start_us":155,"threads_started":1}
I20260925 22:38:30.819113 19442 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:38:30.834345 19442 catalog_manager.cc:1347] Generated new cluster ID: 977f0ffde14d4ea1be1d11c9e1493250
I20260925 22:38:30.834439 19442 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:38:30.844436 19442 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:38:30.846027 19442 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:38:30.857016 19442 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ef6e669f7244e6e85a143e0e9fb5b20: Generated new TSK 0
I20260925 22:38:30.858026 19442 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.246.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.193.keytab.
Entry for principal kudu/127.6.246.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.193.keytab.
I20260925 22:38:31.576607 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.246.212:39789/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.193:0
--local_ip_for_outbound_sockets=127.6.246.193
--webserver_interface=127.6.246.193
--webserver_port=0
--tserver_master_addrs=127.6.246.254:41441
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.193.keytab
--principal=kudu/127.6.246.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:43373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:38:31.715752 19477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:38:31.716163 19477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:38:31.716282 19477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:38:31.721378 19477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:38:31.721518 19477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:38:31.721659 19477 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.193
I20260925 22:38:31.729565 19477 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:43373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.6.246.212:39789/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.193.keytab
--principal=kudu/127.6.246.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.246.193
--webserver_port=0
--tserver_master_addrs=127.6.246.254:41441
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.193
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:38:31.730976 19477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:38:31.732264 19477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:38:31.744341 19484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:38:31.744445 19477 server_base.cc:1034] running on GCE node
W20260925 22:38:31.744851 19485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:38:31.745046 19487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:38:31.745481 19477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:38:31.746167 19477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:38:31.747394 19477 hybrid_clock.cc:648] HybridClock initialized: now 1790375911747363 us; error 42 us; skew 500 ppm
Sep 25 22:38:31 dist-test-slave-wjdr krb5kdc[18920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375911, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:38:31.751204 19477 init.cc:377] Logged in from keytab as kudu/127.6.246.193@KRBTEST.COM (short username kudu)
I20260925 22:38:31.752375 19477 webserver.cc:460] Webserver started at http://127.6.246.193:36487/ using document root <none> and password file <none>
I20260925 22:38:31.753108 19477 fs_manager.cc:359] Metadata directory not provided
I20260925 22:38:31.753198 19477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:38:31.753429 19477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 22:38:31 dist-test-slave-wjdr krb5kdc[18920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375911, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.193@KRBTEST.COM for HTTP/127.6.246.212@KRBTEST.COM
I20260925 22:38:31.771606 19477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-0/data/instance:
uuid: "b8791ac01a8b43f9bc0eb72757c49b4b"
format_stamp: "Formatted at 2026-09-25 22:38:31 on dist-test-slave-wjdr"
server_key: "2eae14c9357f484f381ce766a135690c"
server_key_iv: "647afe646f38a46857d8893e217a53fd"
server_key_version: "kuduclusterkey@0"
I20260925 22:38:31.772338 19477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-0/wal/instance:
uuid: "b8791ac01a8b43f9bc0eb72757c49b4b"
format_stamp: "Formatted at 2026-09-25 22:38:31 on dist-test-slave-wjdr"
server_key: "2eae14c9357f484f381ce766a135690c"
server_key_iv: "647afe646f38a46857d8893e217a53fd"
server_key_version: "kuduclusterkey@0"
I20260925 22:38:31.781162 19477 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.003s sys 0.001s
I20260925 22:38:31.810009 19494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:38:31.811936 19477 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.003s sys 0.001s
I20260925 22:38:31.812091 19477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-0/wal
uuid: "b8791ac01a8b43f9bc0eb72757c49b4b"
format_stamp: "Formatted at 2026-09-25 22:38:31 on dist-test-slave-wjdr"
server_key: "2eae14c9357f484f381ce766a135690c"
server_key_iv: "647afe646f38a46857d8893e217a53fd"
server_key_version: "kuduclusterkey@0"
I20260925 22:38:31.812258 19477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:38:31.832988 19477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:38:31.838886 19477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:38:31.839147 19477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:38:31.839877 19477 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:38:31.840992 19477 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:38:31.841090 19477 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:38:31.841174 19477 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:38:31.841223 19477 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:38:31.851992 19477 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.193:33559
I20260925 22:38:31.853228 19477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
I20260925 22:38:31.863507 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 19477
I20260925 22:38:31.863641 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-0/wal/instance
I20260925 22:38:31.863984 7131 external_mini_cluster.cc:1428] Setting key 04843ee31f5562651236cd4c8b1f4326
I20260925 22:38:31.865916 19609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.193:33559 every 8 connection(s)
WARNING: no policy specified for kudu/127.6.246.194@KRBTEST.COM; defaulting to no policy
Sep 25 22:38:31 dist-test-slave-wjdr krb5kdc[18920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375911, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.193@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.194@KRBTEST.COM" created.
I20260925 22:38:31.908034 19613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:31.870273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.193:43855 (local address 127.6.246.254:41441)
0925 22:38:31.881951 (+ 11678us) server_negotiation.cc:206] Beginning negotiation
0925 22:38:31.881958 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:38:31.881976 (+ 18us) server_negotiation.cc:407] Connection header received
0925 22:38:31.882028 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:38:31.882032 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:38:31.882090 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:38:31.882188 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:38:31.886201 (+ 4013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:31.887064 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:31.894594 (+ 7530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:31.894989 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:31.895074 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:31.900911 (+ 5837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:38:31.900917 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:38:31.900920 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:38:31.900956 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:38:31.903196 (+ 2240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:38:31.905683 (+ 2487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:38:31.905690 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:38:31.905695 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:38:31.905810 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:38:31.906356 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:38:31.906360 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:38:31.906363 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:38:31.906641 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:38:31.906737 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 22:38:31.907574 (+ 837us) server_negotiation.cc:299] Negotiation successful
0925 22:38:31.907730 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11579,"thread_start_us":74,"threads_started":1}
I20260925 22:38:31.910362 19614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:31.870821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:41441 (local address 127.6.246.193:43855)
0925 22:38:31.880061 (+ 9240us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:31.880149 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0925 22:38:31.881420 (+ 1271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:38:31.884746 (+ 3326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:38:31.884758 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:38:31.885180 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:38:31.885982 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:38:31.886004 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:38:31.892749 (+ 6745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:38:31.892754 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:38:31.894390 (+ 1636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:38:31.894408 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:38:31.895320 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:38:31.895323 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:38:31.895541 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:31.896404 (+ 863us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:38:31.896426 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:38:31.900625 (+ 4199us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:38:31.903400 (+ 2775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:38:31.903407 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:38:31.903427 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:38:31.903830 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:38:31.905960 (+ 2130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:38:31.905966 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:38:31.905970 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:38:31.906219 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:38:31.906816 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:38:31.906855 (+ 39us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:38:31.907248 (+ 393us) client_negotiation.cc:769] Sending connection context
0925 22:38:31.908606 (+ 1358us) client_negotiation.cc:240] Negotiation successful
0925 22:38:31.909060 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8912,"thread_start_us":143,"threads_started":1}
I20260925 22:38:31.912281 19610 heartbeater.cc:347] Connected to a master server at 127.6.246.254:41441
I20260925 22:38:31.912568 19610 heartbeater.cc:464] Registering TS with master...
I20260925 22:38:31.913193 19610 heartbeater.cc:511] Master 127.6.246.254:41441 requested a full tablet report, sending...
I20260925 22:38:31.915252 19396 ts_manager.cc:194] Registered new tserver with Master: b8791ac01a8b43f9bc0eb72757c49b4b (127.6.246.193:33559)
I20260925 22:38:31.917121 19396 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.246.193@KRBTEST.COM'} at 127.6.246.193:43855
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.194.keytab.
Entry for principal kudu/127.6.246.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.194.keytab.
I20260925 22:38:31.928859 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.246.212:39789/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.194:0
--local_ip_for_outbound_sockets=127.6.246.194
--webserver_interface=127.6.246.194
--webserver_port=0
--tserver_master_addrs=127.6.246.254:41441
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.194.keytab
--principal=kudu/127.6.246.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:43373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 22:38:31 dist-test-slave-wjdr krb5kdc[18920](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790375911, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 22:38:32.106292 19616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:38:32.106711 19616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:38:32.106840 19616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:38:32.113442 19616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:38:32.113605 19616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:38:32.113746 19616 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.194
I20260925 22:38:32.121984 19616 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:43373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.6.246.212:39789/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.194.keytab
--principal=kudu/127.6.246.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.246.194
--webserver_port=0
--tserver_master_addrs=127.6.246.254:41441
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19616
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.194
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:38:32.123683 19616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:38:32.125139 19616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:38:32.142599 19623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:38:32.143203 19622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:38:32.143806 19625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:38:32.144997 19616 server_base.cc:1034] running on GCE node
I20260925 22:38:32.145634 19616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:38:32.146431 19616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:38:32.147681 19616 hybrid_clock.cc:648] HybridClock initialized: now 1790375912147610 us; error 86 us; skew 500 ppm
Sep 25 22:38:32 dist-test-slave-wjdr krb5kdc[18920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375912, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:38:32.151607 19616 init.cc:377] Logged in from keytab as kudu/127.6.246.194@KRBTEST.COM (short username kudu)
I20260925 22:38:32.152907 19616 webserver.cc:460] Webserver started at http://127.6.246.194:41451/ using document root <none> and password file <none>
I20260925 22:38:32.153627 19616 fs_manager.cc:359] Metadata directory not provided
I20260925 22:38:32.153724 19616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:38:32.154004 19616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 22:38:32 dist-test-slave-wjdr krb5kdc[18920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375912, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.194@KRBTEST.COM for HTTP/127.6.246.212@KRBTEST.COM
I20260925 22:38:32.171808 19616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-1/data/instance:
uuid: "1c5fc0d84b2c4e8fbcf3ca458c75d9b8"
format_stamp: "Formatted at 2026-09-25 22:38:32 on dist-test-slave-wjdr"
server_key: "63edfadca1a111683a7fb0123e46b701"
server_key_iv: "4aea011f1276850961a309b8e7ee39be"
server_key_version: "kuduclusterkey@0"
I20260925 22:38:32.174031 19616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-1/wal/instance:
uuid: "1c5fc0d84b2c4e8fbcf3ca458c75d9b8"
format_stamp: "Formatted at 2026-09-25 22:38:32 on dist-test-slave-wjdr"
server_key: "63edfadca1a111683a7fb0123e46b701"
server_key_iv: "4aea011f1276850961a309b8e7ee39be"
server_key_version: "kuduclusterkey@0"
I20260925 22:38:32.183650 19616 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.001s sys 0.003s
I20260925 22:38:32.221097 19632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:38:32.222981 19616 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.003s sys 0.001s
I20260925 22:38:32.223208 19616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-1/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-1/wal
uuid: "1c5fc0d84b2c4e8fbcf3ca458c75d9b8"
format_stamp: "Formatted at 2026-09-25 22:38:32 on dist-test-slave-wjdr"
server_key: "63edfadca1a111683a7fb0123e46b701"
server_key_iv: "4aea011f1276850961a309b8e7ee39be"
server_key_version: "kuduclusterkey@0"
I20260925 22:38:32.223435 19616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:38:32.239682 19616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:38:32.242807 19616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:38:32.243022 19616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:38:32.243644 19616 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:38:32.244619 19616 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:38:32.244691 19616 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:38:32.244772 19616 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:38:32.244825 19616 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:38:32.259990 19616 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.194:40737
I20260925 22:38:32.261318 19616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
I20260925 22:38:32.267486 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 19616
I20260925 22:38:32.267627 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-1/wal/instance
I20260925 22:38:32.267982 7131 external_mini_cluster.cc:1428] Setting key 49c7d0f68b8b3b4210559a38146c9d2b
I20260925 22:38:32.277411 19745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.194:40737 every 8 connection(s)
WARNING: no policy specified for kudu/127.6.246.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.195@KRBTEST.COM" created.
Sep 25 22:38:32 dist-test-slave-wjdr krb5kdc[18920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375912, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.194@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
I20260925 22:38:32.303210 19613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:32.284058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.194:57427 (local address 127.6.246.254:41441)
0925 22:38:32.284214 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 22:38:32.284219 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:38:32.286808 (+ 2589us) server_negotiation.cc:407] Connection header received
0925 22:38:32.288249 (+ 1441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:38:32.288253 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:38:32.288314 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:38:32.288373 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:38:32.292502 (+ 4129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:32.293344 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:32.295196 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:32.295554 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:32.295633 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:32.298898 (+ 3265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:38:32.298902 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:38:32.298904 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:38:32.298933 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:38:32.300714 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:38:32.301349 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:38:32.301352 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:38:32.301354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:38:32.301440 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:38:32.301864 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:38:32.301868 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:38:32.301870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:38:32.302096 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:38:32.302175 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 22:38:32.302876 (+ 701us) server_negotiation.cc:299] Negotiation successful
0925 22:38:32.303011 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 22:38:32.305437 19749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:32.282900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:41441 (local address 127.6.246.194:57427)
0925 22:38:32.286608 (+ 3708us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:32.286673 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0925 22:38:32.287844 (+ 1171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:38:32.288840 (+ 996us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:38:32.288850 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:38:32.289405 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:38:32.292279 (+ 2874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:38:32.292301 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:38:32.293524 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:38:32.293527 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:38:32.295043 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:38:32.295054 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:38:32.295881 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:38:32.295886 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:38:32.296100 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:32.296841 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:38:32.296864 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:38:32.298668 (+ 1804us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:38:32.300864 (+ 2196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:38:32.300870 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:38:32.300887 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:38:32.301226 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:38:32.301549 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:38:32.301552 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:38:32.301554 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:38:32.301731 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:38:32.302255 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:38:32.302262 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:38:32.302569 (+ 307us) client_negotiation.cc:769] Sending connection context
0925 22:38:32.303968 (+ 1399us) client_negotiation.cc:240] Negotiation successful
0925 22:38:32.304322 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3465,"thread_start_us":158,"threads_started":1}
I20260925 22:38:32.309748 19746 heartbeater.cc:347] Connected to a master server at 127.6.246.254:41441
I20260925 22:38:32.310021 19746 heartbeater.cc:464] Registering TS with master...
I20260925 22:38:32.310619 19746 heartbeater.cc:511] Master 127.6.246.254:41441 requested a full tablet report, sending...
I20260925 22:38:32.311918 19396 ts_manager.cc:194] Registered new tserver with Master: 1c5fc0d84b2c4e8fbcf3ca458c75d9b8 (127.6.246.194:40737)
I20260925 22:38:32.312803 19396 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.246.194@KRBTEST.COM'} at 127.6.246.194:57427
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.195.keytab.
Entry for principal kudu/127.6.246.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.195.keytab.
I20260925 22:38:32.331837 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.246.212:39789/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.195:0
--local_ip_for_outbound_sockets=127.6.246.195
--webserver_interface=127.6.246.195
--webserver_port=0
--tserver_master_addrs=127.6.246.254:41441
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.195.keytab
--principal=kudu/127.6.246.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:43373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 22:38:32 dist-test-slave-wjdr krb5kdc[18920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375911, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.254@KRBTEST.COM for HTTP/127.6.246.212@KRBTEST.COM
W20260925 22:38:32.491832 19751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:38:32.492314 19751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:38:32.492419 19751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:38:32.498394 19751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:38:32.498553 19751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:38:32.498684 19751 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.195
I20260925 22:38:32.506832 19751 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:43373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.6.246.212:39789/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.195.keytab
--principal=kudu/127.6.246.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.246.195
--webserver_port=0
--tserver_master_addrs=127.6.246.254:41441
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.195
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:38:32.508489 19751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:38:32.509842 19751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:38:32.522516 19757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:38:32.523056 19759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:38:32.523852 19751 server_base.cc:1034] running on GCE node
W20260925 22:38:32.524529 19756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:38:32.525061 19751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:38:32.525835 19751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:38:32.527046 19751 hybrid_clock.cc:648] HybridClock initialized: now 1790375912527025 us; error 38 us; skew 500 ppm
Sep 25 22:38:32 dist-test-slave-wjdr krb5kdc[18920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375912, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:38:32.530373 19751 init.cc:377] Logged in from keytab as kudu/127.6.246.195@KRBTEST.COM (short username kudu)
I20260925 22:38:32.531270 19751 webserver.cc:460] Webserver started at http://127.6.246.195:39669/ using document root <none> and password file <none>
I20260925 22:38:32.531843 19751 fs_manager.cc:359] Metadata directory not provided
I20260925 22:38:32.531911 19751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:38:32.532130 19751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 22:38:32 dist-test-slave-wjdr krb5kdc[18920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375912, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.195@KRBTEST.COM for HTTP/127.6.246.212@KRBTEST.COM
I20260925 22:38:32.547338 19751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-2/data/instance:
uuid: "60c3848dcc344688b7916ff475294ee8"
format_stamp: "Formatted at 2026-09-25 22:38:32 on dist-test-slave-wjdr"
server_key: "d2c5d50a5705355612a5d3879013cf60"
server_key_iv: "698c9f7a133014d1e61441fcb11e3469"
server_key_version: "kuduclusterkey@0"
I20260925 22:38:32.548082 19751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-2/wal/instance:
uuid: "60c3848dcc344688b7916ff475294ee8"
format_stamp: "Formatted at 2026-09-25 22:38:32 on dist-test-slave-wjdr"
server_key: "d2c5d50a5705355612a5d3879013cf60"
server_key_iv: "698c9f7a133014d1e61441fcb11e3469"
server_key_version: "kuduclusterkey@0"
I20260925 22:38:32.553342 19751 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 22:38:32.570127 19766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:38:32.571877 19751 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260925 22:38:32.572017 19751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-2/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-2/wal
uuid: "60c3848dcc344688b7916ff475294ee8"
format_stamp: "Formatted at 2026-09-25 22:38:32 on dist-test-slave-wjdr"
server_key: "d2c5d50a5705355612a5d3879013cf60"
server_key_iv: "698c9f7a133014d1e61441fcb11e3469"
server_key_version: "kuduclusterkey@0"
I20260925 22:38:32.572148 19751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:38:32.600875 19751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:38:32.604045 19751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:38:32.604271 19751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:38:32.604947 19751 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:38:32.605924 19751 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:38:32.606000 19751 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:38:32.606092 19751 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:38:32.606148 19751 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:38:32.617518 19751 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.195:36361
I20260925 22:38:32.618459 19751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
I20260925 22:38:32.619408 19879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.195:36361 every 8 connection(s)
I20260925 22:38:32.625751 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 19751
I20260925 22:38:32.625877 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-2/wal/instance
I20260925 22:38:32.626199 7131 external_mini_cluster.cc:1428] Setting key f8efff207d2f1f7c388ff9adba39e54a
Sep 25 22:38:32 dist-test-slave-wjdr krb5kdc[18920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375912, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.195@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
I20260925 22:38:32.633054 19613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:32.621914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.195:38137 (local address 127.6.246.254:41441)
0925 22:38:32.622036 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 22:38:32.622041 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:38:32.622058 (+ 17us) server_negotiation.cc:407] Connection header received
0925 22:38:32.622584 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:38:32.622588 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:38:32.622644 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:38:32.622712 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:38:32.623852 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:32.624400 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:32.625658 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:32.626019 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:32.626118 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:32.629271 (+ 3153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:38:32.629274 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:38:32.629276 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:38:32.629301 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:38:32.630866 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:38:32.631423 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:38:32.631428 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:38:32.631429 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:38:32.631517 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:38:32.631887 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:38:32.631892 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:38:32.631893 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:38:32.632083 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:38:32.632196 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 22:38:32.632770 (+ 574us) server_negotiation.cc:299] Negotiation successful
0925 22:38:32.632894 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 22:38:32.633989 19882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:32.620899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:41441 (local address 127.6.246.195:38137)
0925 22:38:32.621367 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:32.621420 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 22:38:32.622363 (+ 943us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:38:32.622958 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:38:32.622965 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:38:32.623227 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:38:32.623692 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:38:32.623704 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:38:32.624561 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:38:32.624565 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:38:32.625503 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:38:32.625515 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:38:32.626329 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:38:32.626333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:38:32.626531 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:32.627244 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:38:32.627268 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:38:32.629059 (+ 1791us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:38:32.631010 (+ 1951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:38:32.631016 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:38:32.631033 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:38:32.631311 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:38:32.631614 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:38:32.631618 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:38:32.631620 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:38:32.631784 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:38:32.632193 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:38:32.632200 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:38:32.632517 (+ 317us) client_negotiation.cc:769] Sending connection context
0925 22:38:32.632731 (+ 214us) client_negotiation.cc:240] Negotiation successful
0925 22:38:32.633047 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":127,"threads_started":1}
I20260925 22:38:32.635061 19880 heartbeater.cc:347] Connected to a master server at 127.6.246.254:41441
I20260925 22:38:32.635306 19880 heartbeater.cc:464] Registering TS with master...
I20260925 22:38:32.635804 19880 heartbeater.cc:511] Master 127.6.246.254:41441 requested a full tablet report, sending...
I20260925 22:38:32.636874 19396 ts_manager.cc:194] Registered new tserver with Master: 60c3848dcc344688b7916ff475294ee8 (127.6.246.195:36361)
I20260925 22:38:32.637450 19396 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.246.195@KRBTEST.COM'} at 127.6.246.195:38137
I20260925 22:38:32.640617 7131 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:38:32.641305 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 19477
I20260925 22:38:32.648989 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 19616
I20260925 22:38:32.656968 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 19751
I20260925 22:38:32.664640 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 19364
I20260925 22:38:32.671252 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:41441
--webserver_interface=127.6.246.254
--webserver_port=33037
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab
--principal=kudu/127.6.246.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:43373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.246.254:41441
--ranger_config_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.246.212:39789/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.6.246.212:39789/kms with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:38:32.782238 19883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:38:32.782497 19883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:38:32.782537 19883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:38:32.786484 19883 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:38:32.786536 19883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:38:32.786556 19883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:38:32.786571 19883 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:38:32.786586 19883 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:38:32.791996 19883 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:43373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.6.246.212:39789/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.6.246.254:41441
--ranger_config_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab
--principal=kudu/127.6.246.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:41441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.246.254
--webserver_port=33037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19883
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:38:32.793037 19883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:38:32.794005 19883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:38:32.802425 19888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:38:32.802433 19891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:38:32.802632 19883 server_base.cc:1034] running on GCE node
W20260925 22:38:32.802433 19889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:38:32.803349 19883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:38:32.804503 19883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:38:32.805729 19883 hybrid_clock.cc:648] HybridClock initialized: now 1790375912805713 us; error 38 us; skew 500 ppm
Sep 25 22:38:32 dist-test-slave-wjdr krb5kdc[18920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375912, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:38:32.808621 19883 init.cc:377] Logged in from keytab as kudu/127.6.246.254@KRBTEST.COM (short username kudu)
I20260925 22:38:32.809559 19883 webserver.cc:460] Webserver started at http://127.6.246.254:33037/ using document root <none> and password file <none>
I20260925 22:38:32.810115 19883 fs_manager.cc:359] Metadata directory not provided
I20260925 22:38:32.810211 19883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 22:38:32 dist-test-slave-wjdr krb5kdc[18920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375912, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.254@KRBTEST.COM for HTTP/127.6.246.212@KRBTEST.COM
I20260925 22:38:32.871707 19883 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 22:38:32.874207 19899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:38:32.875677 19883 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 22:38:32.875844 19883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/master-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/master-0/wal
uuid: "5ef6e669f7244e6e85a143e0e9fb5b20"
format_stamp: "Formatted at 2026-09-25 22:38:30 on dist-test-slave-wjdr"
server_key: "274c84d3b98c9b5640d7da684e9cbb92"
server_key_iv: "b8e72e075a28b1103712a966ebc06c40"
server_key_version: "kuduclusterkey@0"
I20260925 22:38:32.876446 19883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:38:32.898301 19883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:38:32.901240 19883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:38:32.901456 19883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:38:32.909438 19883 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.254:41441
I20260925 22:38:32.909462 19951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.254:41441 every 8 connection(s)
I20260925 22:38:32.910514 19883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
I20260925 22:38:32.915907 19952 sys_catalog.cc:263] Verifying existing consensus state
I20260925 22:38:32.918179 19952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ef6e669f7244e6e85a143e0e9fb5b20: Bootstrap starting.
I20260925 22:38:32.919483 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 19883
I20260925 22:38:32.919942 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.246.212:39789/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.193:33559
--local_ip_for_outbound_sockets=127.6.246.193
--tserver_master_addrs=127.6.246.254:41441
--webserver_port=36487
--webserver_interface=127.6.246.193
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.193.keytab
--principal=kudu/127.6.246.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:43373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.6.246.212:39789/kms with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:38:32.941005 19952 log.cc:824] T 00000000000000000000000000000000 P 5ef6e669f7244e6e85a143e0e9fb5b20: Log is configured to *not* fsync() on all Append() calls
I20260925 22:38:32.949342 19952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ef6e669f7244e6e85a143e0e9fb5b20: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 22:38:32.949772 19952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ef6e669f7244e6e85a143e0e9fb5b20: Bootstrap complete.
I20260925 22:38:32.954638 19952 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ef6e669f7244e6e85a143e0e9fb5b20 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ef6e669f7244e6e85a143e0e9fb5b20" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 41441 } }
I20260925 22:38:32.955050 19952 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ef6e669f7244e6e85a143e0e9fb5b20 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ef6e669f7244e6e85a143e0e9fb5b20, State: Initialized, Role: FOLLOWER
I20260925 22:38:32.955528 19952 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ef6e669f7244e6e85a143e0e9fb5b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ef6e669f7244e6e85a143e0e9fb5b20" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 41441 } }
I20260925 22:38:32.955670 19952 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ef6e669f7244e6e85a143e0e9fb5b20 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:38:32.955760 19952 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ef6e669f7244e6e85a143e0e9fb5b20 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:38:32.955878 19952 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ef6e669f7244e6e85a143e0e9fb5b20 [term 1 FOLLOWER]: Advancing to term 2
I20260925 22:38:32.957798 19952 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ef6e669f7244e6e85a143e0e9fb5b20 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ef6e669f7244e6e85a143e0e9fb5b20" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 41441 } }
I20260925 22:38:32.958133 19952 leader_election.cc:302] T 00000000000000000000000000000000 P 5ef6e669f7244e6e85a143e0e9fb5b20 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ef6e669f7244e6e85a143e0e9fb5b20; no voters:
I20260925 22:38:32.958455 19952 leader_election.cc:288] T 00000000000000000000000000000000 P 5ef6e669f7244e6e85a143e0e9fb5b20 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 22:38:32.958577 19956 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ef6e669f7244e6e85a143e0e9fb5b20 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 22:38:32.958825 19956 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ef6e669f7244e6e85a143e0e9fb5b20 [term 2 LEADER]: Becoming Leader. State: Replica: 5ef6e669f7244e6e85a143e0e9fb5b20, State: Running, Role: LEADER
I20260925 22:38:32.959187 19956 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ef6e669f7244e6e85a143e0e9fb5b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ef6e669f7244e6e85a143e0e9fb5b20" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 41441 } }
I20260925 22:38:32.959645 19952 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ef6e669f7244e6e85a143e0e9fb5b20 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:38:32.960762 19957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ef6e669f7244e6e85a143e0e9fb5b20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5ef6e669f7244e6e85a143e0e9fb5b20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ef6e669f7244e6e85a143e0e9fb5b20" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 41441 } } }
I20260925 22:38:32.960891 19957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ef6e669f7244e6e85a143e0e9fb5b20 [sys.catalog]: This master's current role is: LEADER
I20260925 22:38:32.961247 19958 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ef6e669f7244e6e85a143e0e9fb5b20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ef6e669f7244e6e85a143e0e9fb5b20. Latest consensus state: current_term: 2 leader_uuid: "5ef6e669f7244e6e85a143e0e9fb5b20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ef6e669f7244e6e85a143e0e9fb5b20" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 41441 } } }
I20260925 22:38:32.961412 19958 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ef6e669f7244e6e85a143e0e9fb5b20 [sys.catalog]: This master's current role is: LEADER
I20260925 22:38:32.961726 19965 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:38:32.962047 19952 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 22:38:32.967643 19965 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:38:32.968719 19965 catalog_manager.cc:1259] Loaded cluster ID: 977f0ffde14d4ea1be1d11c9e1493250
I20260925 22:38:32.970670 19965 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:38:32.973557 19965 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:38:32.975093 19965 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5ef6e669f7244e6e85a143e0e9fb5b20: Loaded TSK: 0
I20260925 22:38:32.976513 19965 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 22:38:33.072772 19954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:38:33.073112 19954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:38:33.073177 19954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:38:33.079125 19954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:38:33.079201 19954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:38:33.079286 19954 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.193
I20260925 22:38:33.086990 19954 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:43373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.6.246.212:39789/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.193.keytab
--principal=kudu/127.6.246.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.193:33559
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.246.193
--webserver_port=36487
--tserver_master_addrs=127.6.246.254:41441
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.193
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:38:33.088481 19954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:38:33.089771 19954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:38:33.101562 19992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:38:33.101815 19990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:38:33.102005 19954 server_base.cc:1034] running on GCE node
W20260925 22:38:33.102557 19989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:38:33.103072 19954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:38:33.103955 19954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:38:33.107215 19954 hybrid_clock.cc:648] HybridClock initialized: now 1790375913107060 us; error 162 us; skew 500 ppm
Sep 25 22:38:33 dist-test-slave-wjdr krb5kdc[18920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375913, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:38:33.114344 19954 init.cc:377] Logged in from keytab as kudu/127.6.246.193@KRBTEST.COM (short username kudu)
I20260925 22:38:33.115304 19954 webserver.cc:460] Webserver started at http://127.6.246.193:36487/ using document root <none> and password file <none>
I20260925 22:38:33.115909 19954 fs_manager.cc:359] Metadata directory not provided
I20260925 22:38:33.116014 19954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 22:38:33 dist-test-slave-wjdr krb5kdc[18920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375913, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.193@KRBTEST.COM for HTTP/127.6.246.212@KRBTEST.COM
I20260925 22:38:33.150321 19954 fs_manager.cc:709] Time spent opening directory manager: real 0.014s user 0.003s sys 0.000s
I20260925 22:38:33.153760 20000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:38:33.155494 19954 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260925 22:38:33.155629 19954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-0/wal
uuid: "b8791ac01a8b43f9bc0eb72757c49b4b"
format_stamp: "Formatted at 2026-09-25 22:38:31 on dist-test-slave-wjdr"
server_key: "2eae14c9357f484f381ce766a135690c"
server_key_iv: "647afe646f38a46857d8893e217a53fd"
server_key_version: "kuduclusterkey@0"
I20260925 22:38:33.156383 19954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:38:33.175192 19954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:38:33.179071 19954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:38:33.179479 19954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:38:33.180505 19954 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:38:33.181774 19954 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:38:33.181900 19954 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:38:33.182035 19954 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:38:33.182123 19954 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:38:33.199568 19954 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.193:33559
I20260925 22:38:33.200947 19954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
I20260925 22:38:33.202054 20113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.193:33559 every 8 connection(s)
I20260925 22:38:33.209928 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 19954
I20260925 22:38:33.210443 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.246.212:39789/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.194:40737
--local_ip_for_outbound_sockets=127.6.246.194
--tserver_master_addrs=127.6.246.254:41441
--webserver_port=41451
--webserver_interface=127.6.246.194
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.194.keytab
--principal=kudu/127.6.246.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:43373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.6.246.212:39789/kms with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 22:38:33 dist-test-slave-wjdr krb5kdc[18920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375913, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.193@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
I20260925 22:38:33.226946 20117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:33.204295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:41441 (local address 127.6.246.193:52867)
0925 22:38:33.205590 (+ 1295us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:33.205648 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 22:38:33.206704 (+ 1056us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:38:33.208880 (+ 2176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:38:33.208889 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:38:33.209250 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:38:33.209876 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:38:33.209897 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:38:33.211892 (+ 1995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:38:33.211896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:38:33.213325 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:38:33.213337 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:38:33.214701 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:38:33.214705 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:38:33.214904 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:33.216604 (+ 1700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:38:33.216626 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:38:33.218490 (+ 1864us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:38:33.223084 (+ 4594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:38:33.223090 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:38:33.223106 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:38:33.223474 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:38:33.223931 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:38:33.223934 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:38:33.223937 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:38:33.224107 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:38:33.224974 (+ 867us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:38:33.224981 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:38:33.225278 (+ 297us) client_negotiation.cc:769] Sending connection context
0925 22:38:33.225469 (+ 191us) client_negotiation.cc:240] Negotiation successful
0925 22:38:33.225835 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1068,"thread_start_us":182,"threads_started":1}
I20260925 22:38:33.229738 20116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:33.204167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.193:52867 (local address 127.6.246.254:41441)
0925 22:38:33.207218 (+ 3051us) server_negotiation.cc:206] Beginning negotiation
0925 22:38:33.207243 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0925 22:38:33.207288 (+ 45us) server_negotiation.cc:407] Connection header received
0925 22:38:33.207578 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:38:33.207592 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:38:33.208008 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:38:33.208361 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:38:33.210080 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:33.211718 (+ 1638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:33.213465 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:33.214298 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:33.214479 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:33.218785 (+ 4306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:38:33.218792 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:38:33.218823 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:38:33.218859 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:38:33.222930 (+ 4071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:38:33.223620 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:38:33.223627 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:38:33.223632 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:38:33.223785 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:38:33.224216 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:38:33.224220 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:38:33.224222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:38:33.224771 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:38:33.224918 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 22:38:33.228486 (+ 3568us) server_negotiation.cc:299] Negotiation successful
0925 22:38:33.228792 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2879,"thread_start_us":137,"threads_started":1}
I20260925 22:38:33.232626 20114 heartbeater.cc:347] Connected to a master server at 127.6.246.254:41441
I20260925 22:38:33.232889 20114 heartbeater.cc:464] Registering TS with master...
I20260925 22:38:33.233482 20114 heartbeater.cc:511] Master 127.6.246.254:41441 requested a full tablet report, sending...
W20260925 22:38:33.235675 20114 heartbeater.cc:644] Failed to heartbeat to 127.6.246.254:41441 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 22:38:33.235802 20114 heartbeater.cc:464] Registering TS with master...
I20260925 22:38:33.235989 20114 heartbeater.cc:511] Master 127.6.246.254:41441 requested a full tablet report, sending...
I20260925 22:38:33.236812 20114 heartbeater.cc:464] Registering TS with master...
I20260925 22:38:33.237007 20114 heartbeater.cc:511] Master 127.6.246.254:41441 requested a full tablet report, sending...
W20260925 22:38:33.237727 20114 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 22:38:33.382975 20118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:38:33.383265 20118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:38:33.383335 20118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:38:33.387687 20118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:38:33.387781 20118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:38:33.387902 20118 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.194
I20260925 22:38:33.394486 20118 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:43373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.6.246.212:39789/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.194.keytab
--principal=kudu/127.6.246.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.194:40737
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.246.194
--webserver_port=41451
--tserver_master_addrs=127.6.246.254:41441
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.194
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:38:33.395861 20118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:38:33.397218 20118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:38:33.415905 20123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:38:33.416057 20124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:38:33.415905 20126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:38:33.417181 20118 server_base.cc:1034] running on GCE node
I20260925 22:38:33.417742 20118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:38:33.418571 20118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:38:33.421887 20118 hybrid_clock.cc:648] HybridClock initialized: now 1790375913421846 us; error 50 us; skew 500 ppm
Sep 25 22:38:33 dist-test-slave-wjdr krb5kdc[18920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375913, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:38:33.425539 20118 init.cc:377] Logged in from keytab as kudu/127.6.246.194@KRBTEST.COM (short username kudu)
I20260925 22:38:33.426700 20118 webserver.cc:460] Webserver started at http://127.6.246.194:41451/ using document root <none> and password file <none>
I20260925 22:38:33.427453 20118 fs_manager.cc:359] Metadata directory not provided
I20260925 22:38:33.427565 20118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 22:38:33 dist-test-slave-wjdr krb5kdc[18920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375913, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.194@KRBTEST.COM for HTTP/127.6.246.212@KRBTEST.COM
I20260925 22:38:33.449222 20118 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 22:38:33.453289 20134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:38:33.454697 20118 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260925 22:38:33.454844 20118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-1/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-1/wal
uuid: "1c5fc0d84b2c4e8fbcf3ca458c75d9b8"
format_stamp: "Formatted at 2026-09-25 22:38:32 on dist-test-slave-wjdr"
server_key: "63edfadca1a111683a7fb0123e46b701"
server_key_iv: "4aea011f1276850961a309b8e7ee39be"
server_key_version: "kuduclusterkey@0"
I20260925 22:38:33.455472 20118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:38:33.484632 20118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:38:33.488562 20118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:38:33.488790 20118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:38:33.489554 20118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:38:33.490684 20118 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:38:33.490751 20118 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:38:33.490819 20118 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:38:33.490850 20118 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:38:33.502840 20118 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.194:40737
I20260925 22:38:33.502918 20248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.194:40737 every 8 connection(s)
I20260925 22:38:33.503906 20118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
I20260925 22:38:33.508706 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 20118
I20260925 22:38:33.509227 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.246.212:39789/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.195:36361
--local_ip_for_outbound_sockets=127.6.246.195
--tserver_master_addrs=127.6.246.254:41441
--webserver_port=39669
--webserver_interface=127.6.246.195
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.195.keytab
--principal=kudu/127.6.246.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:43373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.6.246.212:39789/kms with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 22:38:33 dist-test-slave-wjdr krb5kdc[18920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375913, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.194@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
I20260925 22:38:33.522195 20116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:33.505951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.194:47289 (local address 127.6.246.254:41441)
0925 22:38:33.506096 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 22:38:33.506101 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:38:33.507085 (+ 984us) server_negotiation.cc:407] Connection header received
0925 22:38:33.508142 (+ 1057us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:38:33.508145 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:38:33.508209 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:38:33.508312 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:38:33.510207 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:33.511109 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:33.513023 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:33.513464 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:33.513566 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:33.517165 (+ 3599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:38:33.517171 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:38:33.517177 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:38:33.517216 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:38:33.519155 (+ 1939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:38:33.519931 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:38:33.519937 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:38:33.519940 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:38:33.520039 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:38:33.520575 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:38:33.520581 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:38:33.520586 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:38:33.520868 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:38:33.520970 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 22:38:33.521815 (+ 845us) server_negotiation.cc:299] Negotiation successful
0925 22:38:33.521980 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 22:38:33.524096 20251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:33.506394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:41441 (local address 127.6.246.194:47289)
0925 22:38:33.506898 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:33.506956 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 22:38:33.507902 (+ 946us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:38:33.508543 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:38:33.508554 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:38:33.509017 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:38:33.510015 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:38:33.510037 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:38:33.511306 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:38:33.511311 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:38:33.512863 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:38:33.512878 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:38:33.513844 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:38:33.513850 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:38:33.514106 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:33.514917 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:38:33.514946 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:38:33.516917 (+ 1971us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:38:33.519313 (+ 2396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:38:33.519323 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:38:33.519345 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:38:33.519786 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:38:33.520179 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:38:33.520184 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:38:33.520189 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:38:33.520417 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:38:33.521051 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:38:33.521059 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:38:33.521470 (+ 411us) client_negotiation.cc:769] Sending connection context
0925 22:38:33.522698 (+ 1228us) client_negotiation.cc:240] Negotiation successful
0925 22:38:33.523052 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":154,"threads_started":1}
I20260925 22:38:33.525632 20249 heartbeater.cc:347] Connected to a master server at 127.6.246.254:41441
I20260925 22:38:33.525864 20249 heartbeater.cc:464] Registering TS with master...
I20260925 22:38:33.526415 20249 heartbeater.cc:511] Master 127.6.246.254:41441 requested a full tablet report, sending...
W20260925 22:38:33.528021 20249 heartbeater.cc:644] Failed to heartbeat to 127.6.246.254:41441 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 22:38:33.528134 20249 heartbeater.cc:464] Registering TS with master...
I20260925 22:38:33.528294 20249 heartbeater.cc:511] Master 127.6.246.254:41441 requested a full tablet report, sending...
I20260925 22:38:33.529094 20249 heartbeater.cc:464] Registering TS with master...
I20260925 22:38:33.529247 20249 heartbeater.cc:511] Master 127.6.246.254:41441 requested a full tablet report, sending...
W20260925 22:38:33.530063 20249 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 22:38:33.659865 20252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:38:33.660207 20252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:38:33.660279 20252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:38:33.664994 20252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:38:33.665083 20252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:38:33.665189 20252 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.195
I20260925 22:38:33.671168 20252 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:43373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.6.246.212:39789/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.195.keytab
--principal=kudu/127.6.246.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.195:36361
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.246.195
--webserver_port=39669
--tserver_master_addrs=127.6.246.254:41441
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20252
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.195
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:38:33.672541 20252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:38:33.673703 20252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:38:33.689342 20258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:38:33.689342 20257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:38:33.690397 20260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:38:33.691038 20252 server_base.cc:1034] running on GCE node
I20260925 22:38:33.691569 20252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:38:33.692814 20252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:38:33.694193 20252 hybrid_clock.cc:648] HybridClock initialized: now 1790375913694151 us; error 62 us; skew 500 ppm
Sep 25 22:38:33 dist-test-slave-wjdr krb5kdc[18920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375913, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:38:33.698118 20252 init.cc:377] Logged in from keytab as kudu/127.6.246.195@KRBTEST.COM (short username kudu)
I20260925 22:38:33.699404 20252 webserver.cc:460] Webserver started at http://127.6.246.195:39669/ using document root <none> and password file <none>
I20260925 22:38:33.700196 20252 fs_manager.cc:359] Metadata directory not provided
I20260925 22:38:33.700383 20252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 22:38:33 dist-test-slave-wjdr krb5kdc[18920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375913, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.195@KRBTEST.COM for HTTP/127.6.246.212@KRBTEST.COM
I20260925 22:38:33.730744 20252 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 22:38:33.734719 20268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:38:33.736522 20252 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 22:38:33.736676 20252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-2/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-2/wal
uuid: "60c3848dcc344688b7916ff475294ee8"
format_stamp: "Formatted at 2026-09-25 22:38:32 on dist-test-slave-wjdr"
server_key: "d2c5d50a5705355612a5d3879013cf60"
server_key_iv: "698c9f7a133014d1e61441fcb11e3469"
server_key_version: "kuduclusterkey@0"
I20260925 22:38:33.737440 20252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:38:33.761655 20252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:38:33.765327 20252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:38:33.765538 20252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:38:33.766253 20252 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:38:33.767302 20252 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:38:33.767359 20252 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:38:33.767428 20252 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:38:33.767452 20252 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:38:33.780236 20252 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.195:36361
I20260925 22:38:33.781193 20381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.195:36361 every 8 connection(s)
I20260925 22:38:33.781678 20252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
I20260925 22:38:33.790251 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 20252
Sep 25 22:38:33 dist-test-slave-wjdr krb5kdc[18920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375913, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.195@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
I20260925 22:38:33.797868 20116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:33.785732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.195:54759 (local address 127.6.246.254:41441)
0925 22:38:33.785877 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 22:38:33.785882 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:38:33.785903 (+ 21us) server_negotiation.cc:407] Connection header received
0925 22:38:33.785961 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:38:33.785965 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:38:33.786019 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:38:33.786140 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:38:33.787367 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:33.788199 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:33.789583 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:33.789866 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:33.789989 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:33.793461 (+ 3472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:38:33.793465 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:38:33.793468 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:38:33.793494 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:38:33.795277 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:38:33.795928 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:38:33.795931 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:38:33.795933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:38:33.796013 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:38:33.796412 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:38:33.796415 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:38:33.796417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:38:33.796660 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:38:33.796757 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 22:38:33.797538 (+ 781us) server_negotiation.cc:299] Negotiation successful
0925 22:38:33.797680 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 22:38:33.799671 20384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:33.784248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:41441 (local address 127.6.246.195:54759)
0925 22:38:33.784794 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:33.784859 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0925 22:38:33.785775 (+ 916us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:38:33.786325 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:38:33.786336 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:38:33.786639 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:38:33.787204 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:38:33.787218 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:38:33.788343 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:38:33.788345 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:38:33.789448 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:38:33.789457 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:38:33.790028 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:38:33.790032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:38:33.790216 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:33.791398 (+ 1182us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:38:33.791428 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:38:33.793266 (+ 1838us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:38:33.795408 (+ 2142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:38:33.795416 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:38:33.795436 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:38:33.795830 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:38:33.796109 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:38:33.796111 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:38:33.796113 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:38:33.796303 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:38:33.796839 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:38:33.796851 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:38:33.797217 (+ 366us) client_negotiation.cc:769] Sending connection context
0925 22:38:33.798064 (+ 847us) client_negotiation.cc:240] Negotiation successful
0925 22:38:33.798433 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":169,"threads_started":1}
I20260925 22:38:33.801374 20382 heartbeater.cc:347] Connected to a master server at 127.6.246.254:41441
I20260925 22:38:33.801638 20382 heartbeater.cc:464] Registering TS with master...
I20260925 22:38:33.802206 20382 heartbeater.cc:511] Master 127.6.246.254:41441 requested a full tablet report, sending...
W20260925 22:38:33.803845 20382 heartbeater.cc:644] Failed to heartbeat to 127.6.246.254:41441 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 22:38:33.803982 20382 heartbeater.cc:464] Registering TS with master...
I20260925 22:38:33.804165 20382 heartbeater.cc:511] Master 127.6.246.254:41441 requested a full tablet report, sending...
I20260925 22:38:33.805045 20382 heartbeater.cc:464] Registering TS with master...
I20260925 22:38:33.805217 20382 heartbeater.cc:511] Master 127.6.246.254:41441 requested a full tablet report, sending...
I20260925 22:38:33.805362 20385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:33.791026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43238 (local address 127.6.246.254:41441)
0925 22:38:33.791714 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0925 22:38:33.791720 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:38:33.791732 (+ 12us) server_negotiation.cc:407] Connection header received
0925 22:38:33.791769 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:38:33.791771 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:38:33.791817 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:38:33.791903 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:38:33.793076 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:33.794237 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:33.795835 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:33.796189 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:33.796276 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:33.798273 (+ 1997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:38:33.798276 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:38:33.798279 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:38:33.798313 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:38:33.800972 (+ 2659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:38:33.802762 (+ 1790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:38:33.802765 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:38:33.802767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:38:33.802874 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:38:33.803602 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:38:33.803605 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:38:33.803607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:38:33.803846 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:38:33.804055 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0925 22:38:33.805055 (+ 1000us) server_negotiation.cc:299] Negotiation successful
0925 22:38:33.805190 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":111,"threads_started":1}
W20260925 22:38:33.807030 20382 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 25 22:38:34 dist-test-slave-wjdr krb5kdc[18920](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790375914, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:38:34.237982 20114 heartbeater.cc:464] Registering TS with master...
I20260925 22:38:34.238315 20114 heartbeater.cc:511] Master 127.6.246.254:41441 requested a full tablet report, sending...
I20260925 22:38:34.239527 19916 ts_manager.cc:194] Registered new tserver with Master: b8791ac01a8b43f9bc0eb72757c49b4b (127.6.246.193:33559)
I20260925 22:38:34.241240 19916 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.246.193@KRBTEST.COM'} at 127.6.246.193:52867
I20260925 22:38:34.530304 20249 heartbeater.cc:464] Registering TS with master...
I20260925 22:38:34.530665 20249 heartbeater.cc:511] Master 127.6.246.254:41441 requested a full tablet report, sending...
I20260925 22:38:34.531375 19916 ts_manager.cc:194] Registered new tserver with Master: 1c5fc0d84b2c4e8fbcf3ca458c75d9b8 (127.6.246.194:40737)
I20260925 22:38:34.531997 19916 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.246.194@KRBTEST.COM'} at 127.6.246.194:47289
Sep 25 22:38:34 dist-test-slave-wjdr krb5kdc[18920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375914, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.254@KRBTEST.COM for HTTP/127.6.246.212@KRBTEST.COM
I20260925 22:38:34.807260 20382 heartbeater.cc:464] Registering TS with master...
I20260925 22:38:34.807554 20382 heartbeater.cc:511] Master 127.6.246.254:41441 requested a full tablet report, sending...
I20260925 22:38:34.808320 19916 ts_manager.cc:194] Registered new tserver with Master: 60c3848dcc344688b7916ff475294ee8 (127.6.246.195:36361)
I20260925 22:38:34.809346 19916 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.246.195@KRBTEST.COM'} at 127.6.246.195:54759
I20260925 22:38:34.815809 7131 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:38:34.836562 20403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:34.818138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43244 (local address 127.6.246.254:41441)
0925 22:38:34.818779 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0925 22:38:34.818784 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:38:34.819116 (+ 332us) server_negotiation.cc:407] Connection header received
0925 22:38:34.819160 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:38:34.819163 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:38:34.819211 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:38:34.819301 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:38:34.820202 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:34.821072 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:34.823069 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:34.823423 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:34.823517 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:34.828121 (+ 4604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:38:34.828125 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:38:34.828128 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:38:34.828159 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:38:34.833999 (+ 5840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:38:34.834597 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:38:34.834600 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:38:34.834602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:38:34.834700 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:38:34.835014 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:38:34.835017 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:38:34.835019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:38:34.835239 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:38:34.836096 (+ 857us) server_negotiation.cc:1035] Waiting for connection context
0925 22:38:34.836184 (+ 88us) server_negotiation.cc:299] Negotiation successful
0925 22:38:34.836333 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":88,"threads_started":1}
I20260925 22:38:34.842303 19916 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43244:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:38:34.848308 19916 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
Sep 25 22:38:34 dist-test-slave-wjdr krb5kdc[18920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375914, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.254@KRBTEST.COM for HTTP/127.6.246.212@KRBTEST.COM
I20260925 22:38:34.882716 20413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:34.871595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.193:33559 (local address 127.0.0.1:33260)
0925 22:38:34.876477 (+ 4882us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:34.876491 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:38:34.876688 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:38:34.877548 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:38:34.877554 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:38:34.877581 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:38:34.877746 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:38:34.877753 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:38:34.879258 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:38:34.879264 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:38:34.880715 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:38:34.880727 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:38:34.882141 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:38:34.882145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:38:34.882268 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:34.882313 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 22:38:34.882428 (+ 115us) client_negotiation.cc:240] Negotiation successful
0925 22:38:34.882560 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4800,"thread_start_us":91,"threads_started":1}
I20260925 22:38:34.883193 20414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:34.872019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33260 (local address 127.6.246.193:33559)
0925 22:38:34.877008 (+ 4989us) server_negotiation.cc:206] Beginning negotiation
0925 22:38:34.877016 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 22:38:34.877039 (+ 23us) server_negotiation.cc:407] Connection header received
0925 22:38:34.877117 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:38:34.877122 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:38:34.877357 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:38:34.877515 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:38:34.877920 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:34.879113 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:34.880936 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:34.881800 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:34.881914 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:34.882074 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 22:38:34.882951 (+ 877us) server_negotiation.cc:299] Negotiation successful
0925 22:38:34.883057 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4875,"thread_start_us":109,"threads_started":1}
I20260925 22:38:34.887027 20048 tablet_service.cc:1467] Processing CreateTablet for tablet 37dcd641b4d541758923664cb368d1fc (DEFAULT_TABLE table=test-table [id=d6415ee4b2e74ce7a4772b4081a80712]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:38:34.888293 20048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37dcd641b4d541758923664cb368d1fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:38:34.903738 20411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:34.872920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.195:36361 (local address 127.0.0.1:41660)
0925 22:38:34.878564 (+ 5644us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:34.878577 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:38:34.878657 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:38:34.880099 (+ 1442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:38:34.880102 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:38:34.880120 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:38:34.880268 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:38:34.880275 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:38:34.900618 (+ 20343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:38:34.900622 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:38:34.901807 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:38:34.901833 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:38:34.903261 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:38:34.903265 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:38:34.903363 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:34.903390 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 22:38:34.903469 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 22:38:34.903568 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5585,"thread_start_us":85,"threads_started":1}
I20260925 22:38:34.907896 20416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:34.873102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41660 (local address 127.6.246.195:36361)
0925 22:38:34.877837 (+ 4735us) server_negotiation.cc:206] Beginning negotiation
0925 22:38:34.877844 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:38:34.878750 (+ 906us) server_negotiation.cc:407] Connection header received
0925 22:38:34.878845 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:38:34.878850 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:38:34.879022 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:38:34.879158 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:38:34.881090 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:34.900481 (+ 19391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:34.901997 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:34.902915 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:34.903028 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:34.903188 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 22:38:34.907578 (+ 4390us) server_negotiation.cc:299] Negotiation successful
0925 22:38:34.907720 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4614,"thread_start_us":95,"threads_started":1}
I20260925 22:38:34.911598 20298 tablet_service.cc:1467] Processing CreateTablet for tablet 37dcd641b4d541758923664cb368d1fc (DEFAULT_TABLE table=test-table [id=d6415ee4b2e74ce7a4772b4081a80712]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:38:34.912976 20298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37dcd641b4d541758923664cb368d1fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:38:34.922510 20415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:34.869244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.194:40737 (local address 127.0.0.1:54392)
0925 22:38:34.874300 (+ 5056us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:34.874362 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0925 22:38:34.874493 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:38:34.880420 (+ 5927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:38:34.880423 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:38:34.880440 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:38:34.880545 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:38:34.880550 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:38:34.904043 (+ 23493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:38:34.904047 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:38:34.905253 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:38:34.905265 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:38:34.922033 (+ 16768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:38:34.922039 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:38:34.922169 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:34.922201 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 22:38:34.922264 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 22:38:34.922359 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4862,"thread_start_us":100,"threads_started":1}
I20260925 22:38:34.923391 20412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:34.870355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54392 (local address 127.6.246.194:40737)
0925 22:38:34.878053 (+ 7698us) server_negotiation.cc:206] Beginning negotiation
0925 22:38:34.878060 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:38:34.878081 (+ 21us) server_negotiation.cc:407] Connection header received
0925 22:38:34.878152 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:38:34.878157 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:38:34.878339 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:38:34.880695 (+ 2356us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:38:34.880862 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:34.900103 (+ 19241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:34.905439 (+ 5336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:34.907347 (+ 1908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:34.922886 (+ 15539us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:34.923074 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0925 22:38:34.923138 (+ 64us) server_negotiation.cc:299] Negotiation successful
0925 22:38:34.923250 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7595,"thread_start_us":94,"threads_started":1}
I20260925 22:38:34.927239 20417 tablet_bootstrap.cc:492] T 37dcd641b4d541758923664cb368d1fc P b8791ac01a8b43f9bc0eb72757c49b4b: Bootstrap starting.
I20260925 22:38:34.926903 20167 tablet_service.cc:1467] Processing CreateTablet for tablet 37dcd641b4d541758923664cb368d1fc (DEFAULT_TABLE table=test-table [id=d6415ee4b2e74ce7a4772b4081a80712]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:38:34.928350 20167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37dcd641b4d541758923664cb368d1fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:38:34.931422 20417 tablet_bootstrap.cc:654] T 37dcd641b4d541758923664cb368d1fc P b8791ac01a8b43f9bc0eb72757c49b4b: Neither blocks nor log segments found. Creating new log.
I20260925 22:38:34.932346 20417 log.cc:824] T 37dcd641b4d541758923664cb368d1fc P b8791ac01a8b43f9bc0eb72757c49b4b: Log is configured to *not* fsync() on all Append() calls
I20260925 22:38:34.934741 20417 tablet_bootstrap.cc:492] T 37dcd641b4d541758923664cb368d1fc P b8791ac01a8b43f9bc0eb72757c49b4b: No bootstrap required, opened a new log
I20260925 22:38:34.934969 20417 ts_tablet_manager.cc:1397] T 37dcd641b4d541758923664cb368d1fc P b8791ac01a8b43f9bc0eb72757c49b4b: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260925 22:38:34.938540 20421 tablet_bootstrap.cc:492] T 37dcd641b4d541758923664cb368d1fc P 1c5fc0d84b2c4e8fbcf3ca458c75d9b8: Bootstrap starting.
I20260925 22:38:34.938650 20419 tablet_bootstrap.cc:492] T 37dcd641b4d541758923664cb368d1fc P 60c3848dcc344688b7916ff475294ee8: Bootstrap starting.
I20260925 22:38:34.941119 20421 tablet_bootstrap.cc:654] T 37dcd641b4d541758923664cb368d1fc P 1c5fc0d84b2c4e8fbcf3ca458c75d9b8: Neither blocks nor log segments found. Creating new log.
I20260925 22:38:34.942132 20421 log.cc:824] T 37dcd641b4d541758923664cb368d1fc P 1c5fc0d84b2c4e8fbcf3ca458c75d9b8: Log is configured to *not* fsync() on all Append() calls
I20260925 22:38:34.943496 20419 tablet_bootstrap.cc:654] T 37dcd641b4d541758923664cb368d1fc P 60c3848dcc344688b7916ff475294ee8: Neither blocks nor log segments found. Creating new log.
I20260925 22:38:34.944432 20419 log.cc:824] T 37dcd641b4d541758923664cb368d1fc P 60c3848dcc344688b7916ff475294ee8: Log is configured to *not* fsync() on all Append() calls
I20260925 22:38:34.944551 20421 tablet_bootstrap.cc:492] T 37dcd641b4d541758923664cb368d1fc P 1c5fc0d84b2c4e8fbcf3ca458c75d9b8: No bootstrap required, opened a new log
I20260925 22:38:34.944754 20421 ts_tablet_manager.cc:1397] T 37dcd641b4d541758923664cb368d1fc P 1c5fc0d84b2c4e8fbcf3ca458c75d9b8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260925 22:38:34.945421 20417 raft_consensus.cc:348] T 37dcd641b4d541758923664cb368d1fc P b8791ac01a8b43f9bc0eb72757c49b4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c5fc0d84b2c4e8fbcf3ca458c75d9b8" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 40737 } } peers { permanent_uuid: "b8791ac01a8b43f9bc0eb72757c49b4b" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 33559 } } peers { permanent_uuid: "60c3848dcc344688b7916ff475294ee8" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 36361 } }
I20260925 22:38:34.945705 20417 raft_consensus.cc:374] T 37dcd641b4d541758923664cb368d1fc P b8791ac01a8b43f9bc0eb72757c49b4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:38:34.945752 20417 raft_consensus.cc:729] T 37dcd641b4d541758923664cb368d1fc P b8791ac01a8b43f9bc0eb72757c49b4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8791ac01a8b43f9bc0eb72757c49b4b, State: Initialized, Role: FOLLOWER
I20260925 22:38:34.946275 20417 consensus_queue.cc:260] T 37dcd641b4d541758923664cb368d1fc P b8791ac01a8b43f9bc0eb72757c49b4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c5fc0d84b2c4e8fbcf3ca458c75d9b8" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 40737 } } peers { permanent_uuid: "b8791ac01a8b43f9bc0eb72757c49b4b" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 33559 } } peers { permanent_uuid: "60c3848dcc344688b7916ff475294ee8" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 36361 } }
I20260925 22:38:34.947291 20417 ts_tablet_manager.cc:1428] T 37dcd641b4d541758923664cb368d1fc P b8791ac01a8b43f9bc0eb72757c49b4b: Time spent starting tablet: real 0.012s user 0.004s sys 0.004s
I20260925 22:38:34.947641 20114 heartbeater.cc:503] Master 127.6.246.254:41441 was elected leader, sending a full tablet report...
I20260925 22:38:34.949406 20421 raft_consensus.cc:348] T 37dcd641b4d541758923664cb368d1fc P 1c5fc0d84b2c4e8fbcf3ca458c75d9b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c5fc0d84b2c4e8fbcf3ca458c75d9b8" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 40737 } } peers { permanent_uuid: "b8791ac01a8b43f9bc0eb72757c49b4b" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 33559 } } peers { permanent_uuid: "60c3848dcc344688b7916ff475294ee8" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 36361 } }
I20260925 22:38:34.949749 20421 raft_consensus.cc:374] T 37dcd641b4d541758923664cb368d1fc P 1c5fc0d84b2c4e8fbcf3ca458c75d9b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:38:34.949867 20421 raft_consensus.cc:729] T 37dcd641b4d541758923664cb368d1fc P 1c5fc0d84b2c4e8fbcf3ca458c75d9b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c5fc0d84b2c4e8fbcf3ca458c75d9b8, State: Initialized, Role: FOLLOWER
I20260925 22:38:34.950412 20421 consensus_queue.cc:260] T 37dcd641b4d541758923664cb368d1fc P 1c5fc0d84b2c4e8fbcf3ca458c75d9b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c5fc0d84b2c4e8fbcf3ca458c75d9b8" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 40737 } } peers { permanent_uuid: "b8791ac01a8b43f9bc0eb72757c49b4b" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 33559 } } peers { permanent_uuid: "60c3848dcc344688b7916ff475294ee8" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 36361 } }
I20260925 22:38:34.951552 20421 ts_tablet_manager.cc:1428] T 37dcd641b4d541758923664cb368d1fc P 1c5fc0d84b2c4e8fbcf3ca458c75d9b8: Time spent starting tablet: real 0.007s user 0.003s sys 0.002s
I20260925 22:38:34.951960 20249 heartbeater.cc:503] Master 127.6.246.254:41441 was elected leader, sending a full tablet report...
W20260925 22:38:34.953410 20115 tablet.cc:2369] T 37dcd641b4d541758923664cb368d1fc P b8791ac01a8b43f9bc0eb72757c49b4b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:38:34.955552 20419 tablet_bootstrap.cc:492] T 37dcd641b4d541758923664cb368d1fc P 60c3848dcc344688b7916ff475294ee8: No bootstrap required, opened a new log
I20260925 22:38:34.955785 20419 ts_tablet_manager.cc:1397] T 37dcd641b4d541758923664cb368d1fc P 60c3848dcc344688b7916ff475294ee8: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.003s
I20260925 22:38:34.960875 20419 raft_consensus.cc:348] T 37dcd641b4d541758923664cb368d1fc P 60c3848dcc344688b7916ff475294ee8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c5fc0d84b2c4e8fbcf3ca458c75d9b8" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 40737 } } peers { permanent_uuid: "b8791ac01a8b43f9bc0eb72757c49b4b" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 33559 } } peers { permanent_uuid: "60c3848dcc344688b7916ff475294ee8" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 36361 } }
I20260925 22:38:34.961177 20419 raft_consensus.cc:374] T 37dcd641b4d541758923664cb368d1fc P 60c3848dcc344688b7916ff475294ee8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:38:34.961241 20419 raft_consensus.cc:729] T 37dcd641b4d541758923664cb368d1fc P 60c3848dcc344688b7916ff475294ee8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60c3848dcc344688b7916ff475294ee8, State: Initialized, Role: FOLLOWER
I20260925 22:38:34.961794 20419 consensus_queue.cc:260] T 37dcd641b4d541758923664cb368d1fc P 60c3848dcc344688b7916ff475294ee8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c5fc0d84b2c4e8fbcf3ca458c75d9b8" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 40737 } } peers { permanent_uuid: "b8791ac01a8b43f9bc0eb72757c49b4b" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 33559 } } peers { permanent_uuid: "60c3848dcc344688b7916ff475294ee8" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 36361 } }
I20260925 22:38:34.963037 20419 ts_tablet_manager.cc:1428] T 37dcd641b4d541758923664cb368d1fc P 60c3848dcc344688b7916ff475294ee8: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260925 22:38:34.963429 20382 heartbeater.cc:503] Master 127.6.246.254:41441 was elected leader, sending a full tablet report...
W20260925 22:38:35.005966 20250 tablet.cc:2369] T 37dcd641b4d541758923664cb368d1fc P 1c5fc0d84b2c4e8fbcf3ca458c75d9b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 22:38:35.038328 20383 tablet.cc:2369] T 37dcd641b4d541758923664cb368d1fc P 60c3848dcc344688b7916ff475294ee8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:38:35.198343 20425 raft_consensus.cc:482] T 37dcd641b4d541758923664cb368d1fc P 1c5fc0d84b2c4e8fbcf3ca458c75d9b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:38:35.198567 20425 raft_consensus.cc:504] T 37dcd641b4d541758923664cb368d1fc P 1c5fc0d84b2c4e8fbcf3ca458c75d9b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c5fc0d84b2c4e8fbcf3ca458c75d9b8" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 40737 } } peers { permanent_uuid: "b8791ac01a8b43f9bc0eb72757c49b4b" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 33559 } } peers { permanent_uuid: "60c3848dcc344688b7916ff475294ee8" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 36361 } }
I20260925 22:38:35.199846 20425 leader_election.cc:288] T 37dcd641b4d541758923664cb368d1fc P 1c5fc0d84b2c4e8fbcf3ca458c75d9b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8791ac01a8b43f9bc0eb72757c49b4b (127.6.246.193:33559), 60c3848dcc344688b7916ff475294ee8 (127.6.246.195:36361)
I20260925 22:38:35.205102 20414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:35.200303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.194:39287 (local address 127.6.246.193:33559)
0925 22:38:35.200478 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 22:38:35.200484 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:38:35.200740 (+ 256us) server_negotiation.cc:407] Connection header received
0925 22:38:35.200959 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:38:35.200962 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:38:35.201038 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:38:35.201119 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:38:35.201410 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:35.202131 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:35.204019 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:35.204419 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:35.204590 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:35.204632 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0925 22:38:35.204897 (+ 265us) server_negotiation.cc:299] Negotiation successful
0925 22:38:35.204978 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 22:38:35.205171 20435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:35.200154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.193:33559 (local address 127.6.246.194:39287)
0925 22:38:35.200610 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:35.200644 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 22:38:35.200775 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:38:35.201145 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:38:35.201148 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:38:35.201164 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:38:35.201281 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:38:35.201287 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:38:35.202255 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:38:35.202259 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:38:35.203865 (+ 1606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:38:35.203876 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:38:35.204683 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:38:35.204686 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:38:35.204757 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:35.204774 (+ 17us) client_negotiation.cc:769] Sending connection context
0925 22:38:35.204952 (+ 178us) client_negotiation.cc:240] Negotiation successful
0925 22:38:35.205027 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"mutex_wait_us":248,"thread_start_us":116,"threads_started":1}
I20260925 22:38:35.205495 20416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:35.200323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.194:36243 (local address 127.6.246.195:36361)
0925 22:38:35.200551 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0925 22:38:35.200556 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:38:35.200662 (+ 106us) server_negotiation.cc:407] Connection header received
0925 22:38:35.200810 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:38:35.200814 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:38:35.200874 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:38:35.200955 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:38:35.201275 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:35.202097 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:35.204167 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:35.204835 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:35.204947 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:35.204992 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 22:38:35.205288 (+ 296us) server_negotiation.cc:299] Negotiation successful
0925 22:38:35.205382 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260925 22:38:35.205828 20434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:35.200154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.195:36361 (local address 127.6.246.194:36243)
0925 22:38:35.200552 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:35.200564 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:38:35.200680 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:38:35.200978 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:38:35.200981 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:38:35.200996 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:38:35.201171 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:38:35.201179 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:38:35.202255 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:38:35.202259 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:38:35.203868 (+ 1609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:38:35.203876 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:38:35.205054 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:38:35.205058 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:38:35.205158 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:35.205179 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 22:38:35.205616 (+ 437us) client_negotiation.cc:240] Negotiation successful
0925 22:38:35.205688 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"mutex_wait_us":130,"thread_start_us":135,"threads_started":1}
I20260925 22:38:35.205832 20057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37dcd641b4d541758923664cb368d1fc" candidate_uuid: "1c5fc0d84b2c4e8fbcf3ca458c75d9b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8791ac01a8b43f9bc0eb72757c49b4b" is_pre_election: true
I20260925 22:38:35.206142 20057 raft_consensus.cc:2454] T 37dcd641b4d541758923664cb368d1fc P b8791ac01a8b43f9bc0eb72757c49b4b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c5fc0d84b2c4e8fbcf3ca458c75d9b8 in term 0.
I20260925 22:38:35.206489 20335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37dcd641b4d541758923664cb368d1fc" candidate_uuid: "1c5fc0d84b2c4e8fbcf3ca458c75d9b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60c3848dcc344688b7916ff475294ee8" is_pre_election: true
I20260925 22:38:35.206748 20335 raft_consensus.cc:2454] T 37dcd641b4d541758923664cb368d1fc P 60c3848dcc344688b7916ff475294ee8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c5fc0d84b2c4e8fbcf3ca458c75d9b8 in term 0.
I20260925 22:38:35.206876 20138 leader_election.cc:302] T 37dcd641b4d541758923664cb368d1fc P 1c5fc0d84b2c4e8fbcf3ca458c75d9b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c5fc0d84b2c4e8fbcf3ca458c75d9b8, b8791ac01a8b43f9bc0eb72757c49b4b; no voters:
I20260925 22:38:35.207145 20425 raft_consensus.cc:2781] T 37dcd641b4d541758923664cb368d1fc P 1c5fc0d84b2c4e8fbcf3ca458c75d9b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:38:35.207230 20425 raft_consensus.cc:482] T 37dcd641b4d541758923664cb368d1fc P 1c5fc0d84b2c4e8fbcf3ca458c75d9b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:38:35.207278 20425 raft_consensus.cc:3037] T 37dcd641b4d541758923664cb368d1fc P 1c5fc0d84b2c4e8fbcf3ca458c75d9b8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:38:35.208305 20425 raft_consensus.cc:504] T 37dcd641b4d541758923664cb368d1fc P 1c5fc0d84b2c4e8fbcf3ca458c75d9b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c5fc0d84b2c4e8fbcf3ca458c75d9b8" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 40737 } } peers { permanent_uuid: "b8791ac01a8b43f9bc0eb72757c49b4b" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 33559 } } peers { permanent_uuid: "60c3848dcc344688b7916ff475294ee8" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 36361 } }
I20260925 22:38:35.208700 20425 leader_election.cc:288] T 37dcd641b4d541758923664cb368d1fc P 1c5fc0d84b2c4e8fbcf3ca458c75d9b8 [CANDIDATE]: Term 1 election: Requested vote from peers b8791ac01a8b43f9bc0eb72757c49b4b (127.6.246.193:33559), 60c3848dcc344688b7916ff475294ee8 (127.6.246.195:36361)
I20260925 22:38:35.209014 20057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37dcd641b4d541758923664cb368d1fc" candidate_uuid: "1c5fc0d84b2c4e8fbcf3ca458c75d9b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8791ac01a8b43f9bc0eb72757c49b4b"
I20260925 22:38:35.209016 20335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37dcd641b4d541758923664cb368d1fc" candidate_uuid: "1c5fc0d84b2c4e8fbcf3ca458c75d9b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60c3848dcc344688b7916ff475294ee8"
I20260925 22:38:35.209149 20057 raft_consensus.cc:3037] T 37dcd641b4d541758923664cb368d1fc P b8791ac01a8b43f9bc0eb72757c49b4b [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:38:35.209162 20335 raft_consensus.cc:3037] T 37dcd641b4d541758923664cb368d1fc P 60c3848dcc344688b7916ff475294ee8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:38:35.210100 20057 raft_consensus.cc:2454] T 37dcd641b4d541758923664cb368d1fc P b8791ac01a8b43f9bc0eb72757c49b4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c5fc0d84b2c4e8fbcf3ca458c75d9b8 in term 1.
I20260925 22:38:35.210181 20335 raft_consensus.cc:2454] T 37dcd641b4d541758923664cb368d1fc P 60c3848dcc344688b7916ff475294ee8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c5fc0d84b2c4e8fbcf3ca458c75d9b8 in term 1.
I20260925 22:38:35.210390 20138 leader_election.cc:302] T 37dcd641b4d541758923664cb368d1fc P 1c5fc0d84b2c4e8fbcf3ca458c75d9b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c5fc0d84b2c4e8fbcf3ca458c75d9b8, b8791ac01a8b43f9bc0eb72757c49b4b; no voters:
I20260925 22:38:35.210595 20425 raft_consensus.cc:2781] T 37dcd641b4d541758923664cb368d1fc P 1c5fc0d84b2c4e8fbcf3ca458c75d9b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:38:35.210777 20425 raft_consensus.cc:686] T 37dcd641b4d541758923664cb368d1fc P 1c5fc0d84b2c4e8fbcf3ca458c75d9b8 [term 1 LEADER]: Becoming Leader. State: Replica: 1c5fc0d84b2c4e8fbcf3ca458c75d9b8, State: Running, Role: LEADER
I20260925 22:38:35.211092 20425 consensus_queue.cc:237] T 37dcd641b4d541758923664cb368d1fc P 1c5fc0d84b2c4e8fbcf3ca458c75d9b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c5fc0d84b2c4e8fbcf3ca458c75d9b8" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 40737 } } peers { permanent_uuid: "b8791ac01a8b43f9bc0eb72757c49b4b" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 33559 } } peers { permanent_uuid: "60c3848dcc344688b7916ff475294ee8" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 36361 } }
I20260925 22:38:35.214148 19916 catalog_manager.cc:5495] T 37dcd641b4d541758923664cb368d1fc P 1c5fc0d84b2c4e8fbcf3ca458c75d9b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c5fc0d84b2c4e8fbcf3ca458c75d9b8 (127.6.246.194). New cstate: current_term: 1 leader_uuid: "1c5fc0d84b2c4e8fbcf3ca458c75d9b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c5fc0d84b2c4e8fbcf3ca458c75d9b8" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 40737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8791ac01a8b43f9bc0eb72757c49b4b" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 33559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60c3848dcc344688b7916ff475294ee8" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 36361 } health_report { overall_health: UNKNOWN } } }
I20260925 22:38:35.249672 20412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:35.245736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54398 (local address 127.6.246.194:40737)
0925 22:38:35.245879 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 22:38:35.245884 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:38:35.245899 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:38:35.246080 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:38:35.246083 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:38:35.246151 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:38:35.246264 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:38:35.246515 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:35.247072 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:35.247971 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:35.248209 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:35.248293 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:35.248667 (+ 374us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:38:35.249259 (+ 592us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:38:35.249364 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 22:38:35.249430 (+ 66us) server_negotiation.cc:299] Negotiation successful
0925 22:38:35.249492 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 22:38:35.257402 20057 raft_consensus.cc:1261] T 37dcd641b4d541758923664cb368d1fc P b8791ac01a8b43f9bc0eb72757c49b4b [term 1 FOLLOWER]: Refusing update from remote peer 1c5fc0d84b2c4e8fbcf3ca458c75d9b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:38:35.257402 20335 raft_consensus.cc:1261] T 37dcd641b4d541758923664cb368d1fc P 60c3848dcc344688b7916ff475294ee8 [term 1 FOLLOWER]: Refusing update from remote peer 1c5fc0d84b2c4e8fbcf3ca458c75d9b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:38:35.258292 20425 consensus_queue.cc:1035] T 37dcd641b4d541758923664cb368d1fc P 1c5fc0d84b2c4e8fbcf3ca458c75d9b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8791ac01a8b43f9bc0eb72757c49b4b" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 33559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:38:35.258533 20436 consensus_queue.cc:1035] T 37dcd641b4d541758923664cb368d1fc P 1c5fc0d84b2c4e8fbcf3ca458c75d9b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60c3848dcc344688b7916ff475294ee8" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 36361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:38:35.268234 20441 mvcc.cc:205] Tried to move back new op lower bound from 7333379748884705280 to 7333379748707381248. Current Snapshot: MvccSnapshot[applied={T|T < 7333379748884705280}]
I20260925 22:38:35.278618 19915 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:38:35.278851 19915 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:38:35.281790 19915 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5ef6e669f7244e6e85a143e0e9fb5b20: Sending DeleteTablet for 3 replicas of tablet 37dcd641b4d541758923664cb368d1fc
I20260925 22:38:35.282650 20167 tablet_service.cc:1514] Processing DeleteTablet for tablet 37dcd641b4d541758923664cb368d1fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:38:35 UTC) from {username='kudu', principal='kudu/127.6.246.254@KRBTEST.COM'} at 127.0.0.1:54392
I20260925 22:38:35.282868 20048 tablet_service.cc:1514] Processing DeleteTablet for tablet 37dcd641b4d541758923664cb368d1fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:38:35 UTC) from {username='kudu', principal='kudu/127.6.246.254@KRBTEST.COM'} at 127.0.0.1:33260
I20260925 22:38:35.282905 20298 tablet_service.cc:1514] Processing DeleteTablet for tablet 37dcd641b4d541758923664cb368d1fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:38:35 UTC) from {username='kudu', principal='kudu/127.6.246.254@KRBTEST.COM'} at 127.0.0.1:41660
I20260925 22:38:35.283383 20449 tablet_replica.cc:330] T 37dcd641b4d541758923664cb368d1fc P b8791ac01a8b43f9bc0eb72757c49b4b: stopping tablet replica
I20260925 22:38:35.283833 20449 raft_consensus.cc:2229] T 37dcd641b4d541758923664cb368d1fc P b8791ac01a8b43f9bc0eb72757c49b4b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:38:35.284183 20448 tablet_replica.cc:330] T 37dcd641b4d541758923664cb368d1fc P 1c5fc0d84b2c4e8fbcf3ca458c75d9b8: stopping tablet replica
I20260925 22:38:35.284217 20449 raft_consensus.cc:2258] T 37dcd641b4d541758923664cb368d1fc P b8791ac01a8b43f9bc0eb72757c49b4b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:38:35.284344 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 19954
I20260925 22:38:35.285519 20450 tablet_replica.cc:330] T 37dcd641b4d541758923664cb368d1fc P 60c3848dcc344688b7916ff475294ee8: stopping tablet replica
I20260925 22:38:35.285880 20450 raft_consensus.cc:2229] T 37dcd641b4d541758923664cb368d1fc P 60c3848dcc344688b7916ff475294ee8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:38:35.286156 20448 raft_consensus.cc:2229] T 37dcd641b4d541758923664cb368d1fc P 1c5fc0d84b2c4e8fbcf3ca458c75d9b8 [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:38:35.286218 20450 raft_consensus.cc:2258] T 37dcd641b4d541758923664cb368d1fc P 60c3848dcc344688b7916ff475294ee8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:38:35.286583 20448 raft_consensus.cc:2258] T 37dcd641b4d541758923664cb368d1fc P 1c5fc0d84b2c4e8fbcf3ca458c75d9b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:38:35.288015 20450 ts_tablet_manager.cc:1905] T 37dcd641b4d541758923664cb368d1fc P 60c3848dcc344688b7916ff475294ee8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:38:35.291028 20450 ts_tablet_manager.cc:1918] T 37dcd641b4d541758923664cb368d1fc P 60c3848dcc344688b7916ff475294ee8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:38:35.291146 20450 log.cc:1196] T 37dcd641b4d541758923664cb368d1fc P 60c3848dcc344688b7916ff475294ee8: Deleting WAL directory at /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-2/wal/wals/37dcd641b4d541758923664cb368d1fc
I20260925 22:38:35.291345 20448 ts_tablet_manager.cc:1905] T 37dcd641b4d541758923664cb368d1fc P 1c5fc0d84b2c4e8fbcf3ca458c75d9b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:38:35.291492 20450 ts_tablet_manager.cc:1939] T 37dcd641b4d541758923664cb368d1fc P 60c3848dcc344688b7916ff475294ee8: Deleting consensus metadata
I20260925 22:38:35.292764 19902 catalog_manager.cc:4841] TS 60c3848dcc344688b7916ff475294ee8 (127.6.246.195:36361): tablet 37dcd641b4d541758923664cb368d1fc (table test-table [id=d6415ee4b2e74ce7a4772b4081a80712]) successfully deleted
W20260925 22:38:35.294948 19903 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.246.193:33559 (error 108)
I20260925 22:38:35.295455 20448 ts_tablet_manager.cc:1918] T 37dcd641b4d541758923664cb368d1fc P 1c5fc0d84b2c4e8fbcf3ca458c75d9b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:38:35.295578 20448 log.cc:1196] T 37dcd641b4d541758923664cb368d1fc P 1c5fc0d84b2c4e8fbcf3ca458c75d9b8: Deleting WAL directory at /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790375752797504-7131-0/minicluster-data/ts-1/wal/wals/37dcd641b4d541758923664cb368d1fc
I20260925 22:38:35.295908 20448 ts_tablet_manager.cc:1939] T 37dcd641b4d541758923664cb368d1fc P 1c5fc0d84b2c4e8fbcf3ca458c75d9b8: Deleting consensus metadata
I20260925 22:38:35.295980 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 20118
I20260925 22:38:35.296350 20415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:35.295973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.193:33559 (local address 127.0.0.1:33264)
0925 22:38:35.296193 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:35.296260 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.193:33559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
W20260925 22:38:35.296834 19903 catalog_manager.cc:4584] TS b8791ac01a8b43f9bc0eb72757c49b4b (127.6.246.193:33559): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 37dcd641b4d541758923664cb368d1fc: Network error: Client connection negotiation failed: client connection to 127.6.246.193:33559: connect: Connection refused (error 111)
I20260925 22:38:35.303068 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 20252
I20260925 22:38:35.303372 20415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:35.303079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.194:40737 (local address 127.0.0.1:54408)
0925 22:38:35.303205 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:35.303270 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.194:40737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 22:38:35.309365 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 19883
I20260925 22:38:35.318631 7131 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 22:38:35.664166 7131 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 22:38:35.664317 7131 mini_ranger.cc:66] Stopping Ranger...
I20260925 22:38:36.009918 7131 mini_ranger.cc:68] Stopped Ranger
2026-09-25 22:38:36.010 UTC [18950] LOG: received smart shutdown request
2026-09-25 22:38:36.015 UTC [18950] LOG: background worker "logical replication launcher" (PID 18959) exited with exit code 1
2026-09-25 22:38:36.015 UTC [18954] LOG: shutting down
2026-09-25 22:38:36.193 UTC [18950] LOG: database system is shut down
2026-09-25T22:38:36Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (50120 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790375752797504-7131-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:38:36 dist-test-slave-wjdr krb5kdc[20460](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:38:36 dist-test-slave-wjdr krb5kdc[20460](info): set up 2 sockets
Sep 25 22:38:36 dist-test-slave-wjdr krb5kdc[20460](info): commencing operation
krb5kdc: starting...
W20260925 22:38:37.377305 7131 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Sep 25 22:38:37 dist-test-slave-wjdr krb5kdc[20460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375917, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:38:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:38:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.246.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab.
Entry for principal kudu/127.6.246.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab.
I20260925 22:38:37.509930 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790375752797504-7131-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790375752797504-7131-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790375752797504-7131-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:40285
--webserver_interface=127.6.246.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab
--principal=kudu/127.6.246.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:36705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.246.254:40285,127.6.246.253:39475,127.6.246.252:39919
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:38:37.635284 20473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:38:37.635687 20473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:38:37.635762 20473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:38:37.639971 20473 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:38:37.640085 20473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:38:37.640121 20473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:38:37.640158 20473 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:38:37.640182 20473 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:38:37.645920 20473 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:36705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790375752797504-7131-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790375752797504-7131-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.246.254:40285,127.6.246.253:39475,127.6.246.252:39919
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab
--principal=kudu/127.6.246.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:40285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.246.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790375752797504-7131-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:38:37.647372 20473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:38:37.648711 20473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:38:37.658881 20481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:38:37.658881 20478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:38:37.658918 20479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:38:37.659423 20473 server_base.cc:1034] running on GCE node
I20260925 22:38:37.660095 20473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:38:37.661479 20473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:38:37.662673 20473 hybrid_clock.cc:648] HybridClock initialized: now 1790375917662659 us; error 40 us; skew 500 ppm
Sep 25 22:38:37 dist-test-slave-wjdr krb5kdc[20460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375917, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:38:37.665899 20473 init.cc:377] Logged in from keytab as kudu/127.6.246.254@KRBTEST.COM (short username kudu)
I20260925 22:38:37.666956 20473 webserver.cc:460] Webserver started at http://127.6.246.254:32839/ using document root <none> and password file <none>
I20260925 22:38:37.667580 20473 fs_manager.cc:359] Metadata directory not provided
I20260925 22:38:37.667639 20473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:38:37.667841 20473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:38:37.670264 20473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790375752797504-7131-0/minicluster-data/master-0/data/instance:
uuid: "68d3e3151a514d7b96f9de33facbfca0"
format_stamp: "Formatted at 2026-09-25 22:38:37 on dist-test-slave-wjdr"
server_key: "aa8a94445d9b69016f347353b37e8561"
server_key_iv: "6df3304c46b5792207ab710c256b8eda"
server_key_version: "encryptionkey@0"
I20260925 22:38:37.670874 20473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790375752797504-7131-0/minicluster-data/master-0/wal/instance:
uuid: "68d3e3151a514d7b96f9de33facbfca0"
format_stamp: "Formatted at 2026-09-25 22:38:37 on dist-test-slave-wjdr"
server_key: "aa8a94445d9b69016f347353b37e8561"
server_key_iv: "6df3304c46b5792207ab710c256b8eda"
server_key_version: "encryptionkey@0"
I20260925 22:38:37.675390 20473 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 22:38:37.678153 20488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:38:37.679560 20473 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 22:38:37.679710 20473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790375752797504-7131-0/minicluster-data/master-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790375752797504-7131-0/minicluster-data/master-0/wal
uuid: "68d3e3151a514d7b96f9de33facbfca0"
format_stamp: "Formatted at 2026-09-25 22:38:37 on dist-test-slave-wjdr"
server_key: "aa8a94445d9b69016f347353b37e8561"
server_key_iv: "6df3304c46b5792207ab710c256b8eda"
server_key_version: "encryptionkey@0"
I20260925 22:38:37.679824 20473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790375752797504-7131-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790375752797504-7131-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790375752797504-7131-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:38:37.698238 20473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:38:37.702193 20473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:38:37.702411 20473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:38:37.711067 20473 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.254:40285
I20260925 22:38:37.711088 20540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.254:40285 every 8 connection(s)
I20260925 22:38:37.712301 20473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
I20260925 22:38:37.715981 20541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:38:37.717717 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 20473
I20260925 22:38:37.717885 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790375752797504-7131-0/minicluster-data/master-0/wal/instance
I20260925 22:38:37.718293 7131 external_mini_cluster.cc:1428] Setting key 80a0be6e77b1432b451e59799954af4b
I20260925 22:38:37.722330 20541 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.246.254" port: 40285 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.6.246.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.253@KRBTEST.COM" created.
Sep 25 22:38:37 dist-test-slave-wjdr krb5kdc[20460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375917, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.254@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
I20260925 22:38:37.740423 20544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:37.725740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:40285 (local address 127.0.0.1:51022)
0925 22:38:37.726426 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:37.726476 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 22:38:37.727573 (+ 1097us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:38:37.728376 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:38:37.728382 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:38:37.728920 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:38:37.729202 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:38:37.729217 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:38:37.730636 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:38:37.730639 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:38:37.731409 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:38:37.731417 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:38:37.732083 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:38:37.732088 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:38:37.732184 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:37.733040 (+ 856us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:38:37.733058 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:38:37.734951 (+ 1893us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:38:37.737113 (+ 2162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:38:37.737123 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:38:37.737135 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:38:37.737367 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:38:37.737699 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:38:37.737702 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:38:37.737704 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:38:37.737818 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:38:37.738471 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:38:37.738478 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:38:37.738630 (+ 152us) client_negotiation.cc:769] Sending connection context
0925 22:38:37.738813 (+ 183us) client_negotiation.cc:240] Negotiation successful
0925 22:38:37.739217 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"mutex_wait_us":118,"thread_start_us":188,"threads_started":1}
I20260925 22:38:37.740424 20545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:37.726971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51022 (local address 127.6.246.254:40285)
0925 22:38:37.727287 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0925 22:38:37.727299 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 22:38:37.727351 (+ 52us) server_negotiation.cc:407] Connection header received
0925 22:38:37.727910 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:38:37.727918 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:38:37.728177 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:38:37.728385 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:38:37.729352 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:37.730472 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:37.731526 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:37.731928 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:37.732121 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:37.735156 (+ 3035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:38:37.735162 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:38:37.735174 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:38:37.735196 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:38:37.737005 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:38:37.737477 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:38:37.737483 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:38:37.737486 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:38:37.737601 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:38:37.737905 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:38:37.737910 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:38:37.737912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:38:37.738349 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:38:37.738497 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 22:38:37.738888 (+ 391us) server_negotiation.cc:299] Negotiation successful
0925 22:38:37.739162 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":63,"server-negotiator.queue_time_us":212,"thread_start_us":104,"threads_started":1}
I20260925 22:38:37.743490 20541 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.246.253" port: 39475 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:38:37.744450 20544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:37.744103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.253:39475 (local address 127.0.0.1:40108)
0925 22:38:37.744237 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:37.744353 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.253:39475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 22:38:37.744733 20489 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.246.253:39475: connect: Connection refused (error 111)
I20260925 22:38:37.745641 20544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:37.745439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.253:39475 (local address 127.0.0.1:40124)
0925 22:38:37.745535 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:37.745583 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.253:39475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260925 22:38:37.745805 20541 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.246.253:39475: Network error: Client connection negotiation failed: client connection to 127.6.246.253:39475: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.253.keytab.
Entry for principal kudu/127.6.246.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.253.keytab.
I20260925 22:38:37.747951 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790375752797504-7131-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790375752797504-7131-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790375752797504-7131-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790375752797504-7131-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.253:39475
--webserver_interface=127.6.246.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.253.keytab
--principal=kudu/127.6.246.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:36705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.246.254:40285,127.6.246.253:39475,127.6.246.252:39919
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:38:37.795156 20541 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.246.253" port: 39475 } attempt: 1
I20260925 22:38:37.796435 20544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:37.796050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.253:39475 (local address 127.0.0.1:40132)
0925 22:38:37.796252 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:37.796330 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.253:39475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260925 22:38:37.797331 20544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:37.797075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.253:39475 (local address 127.0.0.1:40144)
0925 22:38:37.797213 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:37.797271 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.253:39475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260925 22:38:37.797523 20541 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.246.253:39475: Network error: Client connection negotiation failed: client connection to 127.6.246.253:39475: connect: Connection refused (error 111)
I20260925 22:38:37.865702 20541 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.246.253" port: 39475 } attempt: 2
I20260925 22:38:37.866858 20544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:37.866469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.253:39475 (local address 127.0.0.1:40152)
0925 22:38:37.866678 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:37.866756 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.253:39475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260925 22:38:37.867693 20544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:37.867442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.253:39475 (local address 127.0.0.1:40156)
0925 22:38:37.867564 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:37.867621 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.253:39475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 22:38:37.867832 20541 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.246.253:39475: Network error: Client connection negotiation failed: client connection to 127.6.246.253:39475: connect: Connection refused (error 111)
W20260925 22:38:37.878425 20548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:38:37.878808 20548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:38:37.878907 20548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:38:37.883226 20548 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:38:37.883318 20548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:38:37.883353 20548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:38:37.883402 20548 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:38:37.883438 20548 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:38:37.889820 20548 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:36705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790375752797504-7131-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790375752797504-7131-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.6.246.254:40285,127.6.246.253:39475,127.6.246.252:39919
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.253.keytab
--principal=kudu/127.6.246.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.253:39475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790375752797504-7131-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.246.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790375752797504-7131-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:38:37.891270 20548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:38:37.892664 20548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:38:37.902777 20556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:38:37.902779 20553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:38:37.902779 20554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:38:37.902935 20548 server_base.cc:1034] running on GCE node
I20260925 22:38:37.903837 20548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:38:37.905297 20548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:38:37.906505 20548 hybrid_clock.cc:648] HybridClock initialized: now 1790375917906487 us; error 38 us; skew 500 ppm
Sep 25 22:38:37 dist-test-slave-wjdr krb5kdc[20460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375917, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:38:37.909724 20548 init.cc:377] Logged in from keytab as kudu/127.6.246.253@KRBTEST.COM (short username kudu)
I20260925 22:38:37.910771 20548 webserver.cc:460] Webserver started at http://127.6.246.253:35815/ using document root <none> and password file <none>
I20260925 22:38:37.911499 20548 fs_manager.cc:359] Metadata directory not provided
I20260925 22:38:37.911597 20548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:38:37.911873 20548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:38:37.914430 20548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790375752797504-7131-0/minicluster-data/master-1/data/instance:
uuid: "774b9fca766248bab60b9732783ec785"
format_stamp: "Formatted at 2026-09-25 22:38:37 on dist-test-slave-wjdr"
server_key: "9baa80fa6461625c27c25a30fc917fc2"
server_key_iv: "8f412bea01d38c57760e1cf8eecff91a"
server_key_version: "encryptionkey@0"
I20260925 22:38:37.915081 20548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790375752797504-7131-0/minicluster-data/master-1/wal/instance:
uuid: "774b9fca766248bab60b9732783ec785"
format_stamp: "Formatted at 2026-09-25 22:38:37 on dist-test-slave-wjdr"
server_key: "9baa80fa6461625c27c25a30fc917fc2"
server_key_iv: "8f412bea01d38c57760e1cf8eecff91a"
server_key_version: "encryptionkey@0"
I20260925 22:38:37.919600 20548 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 22:38:37.922482 20563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:38:37.923902 20548 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:38:37.924068 20548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790375752797504-7131-0/minicluster-data/master-1/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790375752797504-7131-0/minicluster-data/master-1/wal
uuid: "774b9fca766248bab60b9732783ec785"
format_stamp: "Formatted at 2026-09-25 22:38:37 on dist-test-slave-wjdr"
server_key: "9baa80fa6461625c27c25a30fc917fc2"
server_key_iv: "8f412bea01d38c57760e1cf8eecff91a"
server_key_version: "encryptionkey@0"
I20260925 22:38:37.924196 20548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790375752797504-7131-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790375752797504-7131-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790375752797504-7131-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:38:37.943943 20548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:38:37.947651 20548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:38:37.947947 20548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:38:37.956619 20548 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.253:39475
I20260925 22:38:37.956635 20615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.253:39475 every 8 connection(s)
I20260925 22:38:37.957784 20548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790375752797504-7131-0/minicluster-data/master-1/data/info.pb
I20260925 22:38:37.958994 20541 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.246.253" port: 39475 } attempt: 3
I20260925 22:38:37.961742 20616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:38:37.966131 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 20548
I20260925 22:38:37.966303 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790375752797504-7131-0/minicluster-data/master-1/wal/instance
I20260925 22:38:37.966639 7131 external_mini_cluster.cc:1428] Setting key b180aad04e4b48760de8701ad6bb55e8
I20260925 22:38:37.966817 20616 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.246.254" port: 40285 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 22:38:37 dist-test-slave-wjdr krb5kdc[20460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375917, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.254@KRBTEST.COM for kudu/127.6.246.253@KRBTEST.COM
I20260925 22:38:37.973945 20544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:37.959686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.253:39475 (local address 127.0.0.1:40158)
0925 22:38:37.959852 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:37.959872 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 22:38:37.959986 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:38:37.962812 (+ 2826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:38:37.962816 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:38:37.963062 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:38:37.963238 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:38:37.963247 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:38:37.964702 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:38:37.964706 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:38:37.965317 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:38:37.965330 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:38:37.966143 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:38:37.966148 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:38:37.966277 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:37.966944 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:38:37.966958 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:38:37.968391 (+ 1433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:38:37.971510 (+ 3119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:38:37.971514 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:38:37.971520 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:38:37.971828 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:38:37.972254 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:38:37.972258 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:38:37.972260 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:38:37.972363 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:38:37.973284 (+ 921us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:38:37.973288 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:38:37.973399 (+ 111us) client_negotiation.cc:769] Sending connection context
0925 22:38:37.973515 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 22:38:37.973685 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260925 22:38:37.975503 20618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:37.960625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40158 (local address 127.6.246.253:39475)
0925 22:38:37.961120 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0925 22:38:37.961137 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 22:38:37.961180 (+ 43us) server_negotiation.cc:407] Connection header received
0925 22:38:37.962020 (+ 840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:38:37.962058 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:38:37.962472 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:38:37.962862 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:38:37.963395 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:37.964557 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:37.965456 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:37.965984 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:37.966179 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:37.968662 (+ 2483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:38:37.968668 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:38:37.968683 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:38:37.968712 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:38:37.971397 (+ 2685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:38:37.971963 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:38:37.971973 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:38:37.971982 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:38:37.972139 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:38:37.972501 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:38:37.972508 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:38:37.972511 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:38:37.973156 (+ 645us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:38:37.973312 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 22:38:37.973729 (+ 417us) server_negotiation.cc:299] Negotiation successful
0925 22:38:37.974138 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":155,"threads_started":1}
Sep 25 22:38:37 dist-test-slave-wjdr krb5kdc[20460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375917, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.253@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.246.252@KRBTEST.COM; defaulting to no policy
I20260925 22:38:37.978201 20541 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.246.252" port: 39919 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:38:37.979153 20544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:37.978806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.252:39919 (local address 127.0.0.1:38978)
0925 22:38:37.978986 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:37.979054 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.252:39919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260925 22:38:37.979970 20544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:37.979662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.252:39919 (local address 127.0.0.1:38994)
0925 22:38:37.979843 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:37.979895 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.252:39919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.252@KRBTEST.COM" created.
W20260925 22:38:37.980186 20541 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.246.252:39919: Network error: Client connection negotiation failed: client connection to 127.6.246.252:39919: connect: Connection refused (error 111)
I20260925 22:38:37.981060 20620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:37.969442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:40285 (local address 127.0.0.1:51026)
0925 22:38:37.970190 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:37.970257 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0925 22:38:37.970435 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:38:37.970738 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:38:37.970745 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:38:37.971172 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:38:37.971317 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:38:37.971323 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:38:37.972824 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:38:37.972828 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:38:37.973666 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:38:37.973676 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:38:37.974450 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:38:37.974453 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:38:37.974585 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:37.975179 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:38:37.975190 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:38:37.976787 (+ 1597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:38:37.979024 (+ 2237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:38:37.979035 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:38:37.979041 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:38:37.979385 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:38:37.979789 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:38:37.979793 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:38:37.979796 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:38:37.979897 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:38:37.980423 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:38:37.980433 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:38:37.980605 (+ 172us) client_negotiation.cc:769] Sending connection context
0925 22:38:37.980740 (+ 135us) client_negotiation.cc:240] Negotiation successful
0925 22:38:37.980898 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":178,"threads_started":1}
I20260925 22:38:37.981176 20545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:37.969510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51026 (local address 127.6.246.254:40285)
0925 22:38:37.969674 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 22:38:37.969680 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:38:37.970338 (+ 658us) server_negotiation.cc:407] Connection header received
0925 22:38:37.970555 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:38:37.970558 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:38:37.970617 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:38:37.970703 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:38:37.971447 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:37.972718 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:37.973834 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:37.974198 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:37.974280 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:37.976944 (+ 2664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:38:37.976951 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:38:37.976956 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:38:37.976990 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:38:37.978910 (+ 1920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:38:37.979522 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:38:37.979528 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:38:37.979530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:38:37.979683 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:38:37.980051 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:38:37.980057 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:38:37.980061 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:38:37.980320 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:38:37.980402 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 22:38:37.980834 (+ 432us) server_negotiation.cc:299] Negotiation successful
0925 22:38:37.980978 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 22:38:37.982168 20616 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.246.253" port: 39475 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 22:38:37 dist-test-slave-wjdr krb5kdc[20460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375917, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.253@KRBTEST.COM for kudu/127.6.246.253@KRBTEST.COM
I20260925 22:38:37.990533 20620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:37.982670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.253:39475 (local address 127.0.0.1:40166)
0925 22:38:37.982840 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:37.982855 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:38:37.982937 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:38:37.983165 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:38:37.983168 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:38:37.983344 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:38:37.983450 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:38:37.983456 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:38:37.984176 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:38:37.984182 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:38:37.984837 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:38:37.984846 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:38:37.985360 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:38:37.985364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:38:37.985513 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:37.985997 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:38:37.986010 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:38:37.987369 (+ 1359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:38:37.989126 (+ 1757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:38:37.989131 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:38:37.989134 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:38:37.989390 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:38:37.989666 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:38:37.989668 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:38:37.989670 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:38:37.989745 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:38:37.990083 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:38:37.990085 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:38:37.990173 (+ 88us) client_negotiation.cc:769] Sending connection context
0925 22:38:37.990258 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 22:38:37.990365 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":33}
I20260925 22:38:37.990540 20618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:37.982747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40166 (local address 127.6.246.253:39475)
0925 22:38:37.982845 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0925 22:38:37.982852 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:38:37.982897 (+ 45us) server_negotiation.cc:407] Connection header received
0925 22:38:37.983033 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:38:37.983035 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:38:37.983073 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:38:37.983157 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:38:37.983552 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:37.984077 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:37.984951 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:37.985256 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:37.985362 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:37.987527 (+ 2165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:38:37.987530 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:38:37.987533 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:38:37.987553 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:38:37.989013 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:38:37.989500 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:38:37.989503 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:38:37.989504 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:38:37.989576 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:38:37.989820 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:38:37.989822 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:38:37.989823 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:38:37.989993 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:38:37.990093 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 22:38:37.990314 (+ 221us) server_negotiation.cc:299] Negotiation successful
0925 22:38:37.990416 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":35}
I20260925 22:38:37.991856 20616 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.246.252" port: 39919 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:38:37.992651 20620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:37.992347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.252:39919 (local address 127.0.0.1:39008)
0925 22:38:37.992485 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:37.992586 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.252:39919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 22:38:37.992873 20567 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.246.252:39919: connect: Connection refused (error 111)
I20260925 22:38:37.993914 20620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:37.993707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.252:39919 (local address 127.0.0.1:39020)
0925 22:38:37.993822 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:37.993865 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.252:39919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 22:38:37.994048 20616 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.246.252:39919: Network error: Client connection negotiation failed: client connection to 127.6.246.252:39919: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.252.keytab.
Entry for principal kudu/127.6.246.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.252.keytab.
I20260925 22:38:37.998260 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790375752797504-7131-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790375752797504-7131-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790375752797504-7131-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790375752797504-7131-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.252:39919
--webserver_interface=127.6.246.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.252.keytab
--principal=kudu/127.6.246.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:36705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.246.254:40285,127.6.246.253:39475,127.6.246.252:39919
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:38:38.011358 20541 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.246.252" port: 39919 } attempt: 1
I20260925 22:38:38.012614 20544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:38.012168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.252:39919 (local address 127.0.0.1:39022)
0925 22:38:38.012374 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:38.012447 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.252:39919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260925 22:38:38.013386 20544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:38.013188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.252:39919 (local address 127.0.0.1:39036)
0925 22:38:38.013292 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:38.013337 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.252:39919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260925 22:38:38.013514 20541 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.246.252:39919: Network error: Client connection negotiation failed: client connection to 127.6.246.252:39919: connect: Connection refused (error 111)
I20260925 22:38:38.043396 20616 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.246.252" port: 39919 } attempt: 1
I20260925 22:38:38.044974 20620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:38.044413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.252:39919 (local address 127.0.0.1:39042)
0925 22:38:38.044683 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:38.044820 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.252:39919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260925 22:38:38.046257 20620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:38.045939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.252:39919 (local address 127.0.0.1:39044)
0925 22:38:38.046080 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:38.046157 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.252:39919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 22:38:38.046552 20616 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.246.252:39919: Network error: Client connection negotiation failed: client connection to 127.6.246.252:39919: connect: Connection refused (error 111)
I20260925 22:38:38.088701 20541 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.246.252" port: 39919 } attempt: 2
I20260925 22:38:38.089893 20544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:38.089476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.252:39919 (local address 127.0.0.1:39052)
0925 22:38:38.089682 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:38.089774 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.252:39919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260925 22:38:38.090782 20544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:38.090500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.252:39919 (local address 127.0.0.1:39054)
0925 22:38:38.090638 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:38.090702 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.252:39919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260925 22:38:38.090934 20541 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.246.252:39919: Network error: Client connection negotiation failed: client connection to 127.6.246.252:39919: connect: Connection refused (error 111)
I20260925 22:38:38.114756 20616 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.246.252" port: 39919 } attempt: 2
I20260925 22:38:38.115878 20620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:38.115528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.252:39919 (local address 127.0.0.1:39066)
0925 22:38:38.115703 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:38.115780 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.252:39919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260925 22:38:38.116627 20620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:38.116390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.252:39919 (local address 127.0.0.1:39082)
0925 22:38:38.116514 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:38.116562 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.252:39919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260925 22:38:38.116760 20616 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.246.252:39919: Network error: Client connection negotiation failed: client connection to 127.6.246.252:39919: connect: Connection refused (error 111)
W20260925 22:38:38.135995 20623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:38:38.136365 20623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:38:38.136444 20623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:38:38.140753 20623 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:38:38.140839 20623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:38:38.140877 20623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:38:38.140930 20623 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:38:38.140965 20623 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:38:38.146835 20623 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:36705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790375752797504-7131-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790375752797504-7131-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.6.246.254:40285,127.6.246.253:39475,127.6.246.252:39919
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.252.keytab
--principal=kudu/127.6.246.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.252:39919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790375752797504-7131-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.246.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20623
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790375752797504-7131-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:38:38.148383 20623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:38:38.149646 20623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:38:38.159401 20628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:38:38.159396 20631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:38:38.159581 20623 server_base.cc:1034] running on GCE node
W20260925 22:38:38.159396 20629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:38:38.160243 20623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:38:38.161602 20623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:38:38.162834 20623 hybrid_clock.cc:648] HybridClock initialized: now 1790375918162820 us; error 30 us; skew 500 ppm
Sep 25 22:38:38 dist-test-slave-wjdr krb5kdc[20460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375918, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:38:38.170432 20623 init.cc:377] Logged in from keytab as kudu/127.6.246.252@KRBTEST.COM (short username kudu)
I20260925 22:38:38.171605 20623 webserver.cc:460] Webserver started at http://127.6.246.252:43687/ using document root <none> and password file <none>
I20260925 22:38:38.172297 20623 fs_manager.cc:359] Metadata directory not provided
I20260925 22:38:38.172384 20623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:38:38.172679 20623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:38:38.175132 20623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790375752797504-7131-0/minicluster-data/master-2/data/instance:
uuid: "8aea6bd8085c4ff6998e58e4157d8ef2"
format_stamp: "Formatted at 2026-09-25 22:38:38 on dist-test-slave-wjdr"
server_key: "91b8eef287864a0d853d1956a459ba02"
server_key_iv: "638ffaab40e397b5af5f99b692d2e847"
server_key_version: "encryptionkey@0"
I20260925 22:38:38.175818 20623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790375752797504-7131-0/minicluster-data/master-2/wal/instance:
uuid: "8aea6bd8085c4ff6998e58e4157d8ef2"
format_stamp: "Formatted at 2026-09-25 22:38:38 on dist-test-slave-wjdr"
server_key: "91b8eef287864a0d853d1956a459ba02"
server_key_iv: "638ffaab40e397b5af5f99b692d2e847"
server_key_version: "encryptionkey@0"
I20260925 22:38:38.180333 20623 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260925 22:38:38.183259 20638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:38:38.184865 20623 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:38:38.185027 20623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790375752797504-7131-0/minicluster-data/master-2/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790375752797504-7131-0/minicluster-data/master-2/wal
uuid: "8aea6bd8085c4ff6998e58e4157d8ef2"
format_stamp: "Formatted at 2026-09-25 22:38:38 on dist-test-slave-wjdr"
server_key: "91b8eef287864a0d853d1956a459ba02"
server_key_iv: "638ffaab40e397b5af5f99b692d2e847"
server_key_version: "encryptionkey@0"
I20260925 22:38:38.185161 20623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790375752797504-7131-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790375752797504-7131-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790375752797504-7131-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:38:38.190111 20541 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.246.252" port: 39919 } attempt: 3
I20260925 22:38:38.191145 20544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:38.190804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.252:39919 (local address 127.0.0.1:39088)
0925 22:38:38.190967 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:38.191034 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.252:39919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260925 22:38:38.191903 20544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:38.191684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.252:39919 (local address 127.0.0.1:39094)
0925 22:38:38.191791 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:38.191843 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.252:39919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260925 22:38:38.192039 20541 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.246.252:39919: Network error: Client connection negotiation failed: client connection to 127.6.246.252:39919: connect: Connection refused (error 111)
I20260925 22:38:38.198151 20623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:38:38.204304 20623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:38:38.204842 20623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:38:38.207907 20616 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.246.252" port: 39919 } attempt: 3
I20260925 22:38:38.209143 20620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:38.208679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.252:39919 (local address 127.0.0.1:39108)
0925 22:38:38.208937 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:38.209022 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.252:39919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260925 22:38:38.210214 20620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:38.209907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.252:39919 (local address 127.0.0.1:39120)
0925 22:38:38.210090 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:38.210146 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.252:39919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
W20260925 22:38:38.210459 20616 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.246.252:39919: Network error: Client connection negotiation failed: client connection to 127.6.246.252:39919: connect: Connection refused (error 111)
I20260925 22:38:38.216179 20623 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.252:39919
I20260925 22:38:38.216279 20690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.252:39919 every 8 connection(s)
I20260925 22:38:38.217468 20623 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790375752797504-7131-0/minicluster-data/master-2/data/info.pb
I20260925 22:38:38.222046 20691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:38:38.227636 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 20623
I20260925 22:38:38.227792 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790375752797504-7131-0/minicluster-data/master-2/wal/instance
I20260925 22:38:38.228137 7131 external_mini_cluster.cc:1428] Setting key bb92c4d8adac6027af17337c8e739028
I20260925 22:38:38.228230 20691 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.246.254" port: 40285 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 22:38:38 dist-test-slave-wjdr krb5kdc[20460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375917, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.246.252@KRBTEST.COM
Sep 25 22:38:38 dist-test-slave-wjdr krb5kdc[20460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375918, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.252@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
I20260925 22:38:38.244941 20545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:38.230855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51030 (local address 127.6.246.254:40285)
0925 22:38:38.231023 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 22:38:38.231029 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:38:38.231482 (+ 453us) server_negotiation.cc:407] Connection header received
0925 22:38:38.231904 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:38:38.231908 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:38:38.231968 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:38:38.232069 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:38:38.233364 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:38.234315 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:38.235846 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:38.236236 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:38.236368 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:38.240531 (+ 4163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:38:38.240538 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:38:38.240543 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:38:38.240595 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:38:38.242566 (+ 1971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:38:38.243121 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:38:38.243126 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:38:38.243131 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:38:38.243219 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:38:38.243510 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:38:38.243516 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:38:38.243521 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:38:38.243823 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:38:38.243938 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 22:38:38.244526 (+ 588us) server_negotiation.cc:299] Negotiation successful
0925 22:38:38.244695 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 22:38:38.245693 20694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:38.229912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39122 (local address 127.6.246.252:39919)
0925 22:38:38.230373 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0925 22:38:38.230394 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 22:38:38.230442 (+ 48us) server_negotiation.cc:407] Connection header received
0925 22:38:38.231440 (+ 998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:38:38.231466 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:38:38.231762 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:38:38.231964 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:38:38.232864 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:38.234025 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:38.234886 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:38.235385 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:38.235551 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:38.239791 (+ 4240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:38:38.239800 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:38:38.239818 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:38:38.239847 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:38:38.242077 (+ 2230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:38:38.242614 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:38:38.242620 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:38:38.242628 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:38:38.242780 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:38:38.243092 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:38:38.243096 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:38:38.243098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:38:38.243715 (+ 617us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:38:38.243946 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0925 22:38:38.244217 (+ 271us) server_negotiation.cc:299] Negotiation successful
0925 22:38:38.244598 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":89,"server-negotiator.queue_time_us":300,"thread_start_us":129,"threads_started":1}
I20260925 22:38:38.245756 20695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:38.230686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:40285 (local address 127.0.0.1:51030)
0925 22:38:38.231315 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:38.231369 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 22:38:38.231684 (+ 315us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:38:38.232131 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:38:38.232137 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:38:38.232684 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:38:38.233167 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:38:38.233188 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:38:38.234501 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:38:38.234506 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:38:38.235708 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:38:38.235723 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:38:38.236407 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:38:38.236411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:38:38.236568 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:38.237174 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:38:38.237198 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:38:38.240303 (+ 3105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:38:38.242678 (+ 2375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:38:38.242690 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:38:38.242698 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:38:38.242980 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:38:38.243341 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:38:38.243344 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:38:38.243346 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:38:38.243425 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:38:38.244003 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:38:38.244014 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:38:38.244217 (+ 203us) client_negotiation.cc:769] Sending connection context
0925 22:38:38.245210 (+ 993us) client_negotiation.cc:240] Negotiation successful
0925 22:38:38.245458 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"mutex_wait_us":1,"spinlock_wait_cycles":1280,"thread_start_us":170,"threads_started":1}
I20260925 22:38:38.247459 20691 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.246.253" port: 39475 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 22:38:38 dist-test-slave-wjdr krb5kdc[20460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375918, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.252@KRBTEST.COM for kudu/127.6.246.253@KRBTEST.COM
Sep 25 22:38:38 dist-test-slave-wjdr krb5kdc[20460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375917, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
I20260925 22:38:38.258173 20618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:38.248260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40180 (local address 127.6.246.253:39475)
0925 22:38:38.248425 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 22:38:38.248432 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:38:38.248536 (+ 104us) server_negotiation.cc:407] Connection header received
0925 22:38:38.248782 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:38:38.248787 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:38:38.248862 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:38:38.248931 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:38:38.250245 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:38.251079 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:38.252128 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:38.252365 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:38.252533 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:38.254646 (+ 2113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:38:38.254650 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:38:38.254652 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:38:38.254678 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:38:38.256380 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:38:38.256866 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:38:38.256869 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:38:38.256871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:38:38.256947 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:38:38.257240 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:38:38.257246 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:38:38.257249 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:38:38.257509 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:38:38.257614 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 22:38:38.257890 (+ 276us) server_negotiation.cc:299] Negotiation successful
0925 22:38:38.258002 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 22:38:38.258190 20695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:38.248059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.253:39475 (local address 127.0.0.1:40180)
0925 22:38:38.248425 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:38.248441 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:38:38.248640 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:38:38.249687 (+ 1047us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:38:38.249692 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:38:38.249925 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:38:38.250071 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:38:38.250079 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:38:38.251249 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:38:38.251254 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:38:38.251995 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:38:38.252004 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:38:38.252505 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:38:38.252509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:38:38.252597 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:38.253126 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:38:38.253137 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:38:38.254485 (+ 1348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:38:38.256507 (+ 2022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:38:38.256510 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:38:38.256513 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:38:38.256771 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:38:38.257033 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:38:38.257037 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:38:38.257041 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:38:38.257134 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:38:38.257606 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:38:38.257610 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:38:38.257704 (+ 94us) client_negotiation.cc:769] Sending connection context
0925 22:38:38.257809 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 22:38:38.257953 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294}
I20260925 22:38:38.259101 20691 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.246.252" port: 39919 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:38:38.259245 20545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:38.249158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51044 (local address 127.6.246.254:40285)
0925 22:38:38.249303 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 22:38:38.249309 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:38:38.249324 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:38:38.249365 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:38:38.249368 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:38:38.249421 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:38:38.249511 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:38:38.250271 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:38.250863 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:38.251981 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:38.252216 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:38.252329 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:38.255203 (+ 2874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:38:38.255207 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:38:38.255210 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:38:38.255235 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:38:38.256978 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:38:38.257435 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:38:38.257437 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:38:38.257439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:38:38.257509 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:38:38.257785 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:38:38.257790 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:38:38.257793 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:38:38.258012 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:38:38.258293 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0925 22:38:38.258813 (+ 520us) server_negotiation.cc:299] Negotiation successful
0925 22:38:38.259089 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Sep 25 22:38:38 dist-test-slave-wjdr krb5kdc[20460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375918, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.252@KRBTEST.COM for kudu/127.6.246.252@KRBTEST.COM
Sep 25 22:38:38 dist-test-slave-wjdr krb5kdc[20460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375917, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.246.253@KRBTEST.COM
I20260925 22:38:38.268044 20695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:38.259694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.252:39919 (local address 127.0.0.1:39136)
0925 22:38:38.259812 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:38.259825 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:38:38.259907 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:38:38.260226 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:38:38.260229 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:38:38.260400 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:38:38.260517 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:38:38.260526 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:38:38.261251 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:38:38.261254 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:38:38.261765 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:38:38.261771 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:38:38.262214 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:38:38.262217 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:38:38.262319 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:38.262799 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:38:38.262812 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:38:38.264054 (+ 1242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:38:38.266509 (+ 2455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:38:38.266511 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:38:38.266513 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:38:38.266704 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:38:38.266989 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:38:38.266992 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:38:38.266993 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:38:38.267062 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:38:38.267463 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:38:38.267466 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:38:38.267575 (+ 109us) client_negotiation.cc:769] Sending connection context
0925 22:38:38.267670 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 22:38:38.267815 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":18}
I20260925 22:38:38.268059 20694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:38.259968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39136 (local address 127.6.246.252:39919)
0925 22:38:38.260060 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0925 22:38:38.260064 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:38:38.260075 (+ 11us) server_negotiation.cc:407] Connection header received
0925 22:38:38.260105 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:38:38.260107 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:38:38.260138 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:38:38.260216 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:38:38.260614 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:38.261142 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:38.261870 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:38.262114 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:38.262218 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:38.264172 (+ 1954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:38:38.264176 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:38:38.264179 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:38:38.264199 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:38:38.266286 (+ 2087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:38:38.266800 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:38:38.266804 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:38:38.266805 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:38:38.266891 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:38:38.267142 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:38:38.267146 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:38:38.267148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:38:38.267372 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:38:38.267446 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 22:38:38.267771 (+ 325us) server_negotiation.cc:299] Negotiation successful
0925 22:38:38.267876 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":31}
I20260925 22:38:38.269596 20618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:38.260565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40184 (local address 127.6.246.253:39475)
0925 22:38:38.260682 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0925 22:38:38.260686 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:38:38.260740 (+ 54us) server_negotiation.cc:407] Connection header received
0925 22:38:38.260885 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:38:38.260888 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:38:38.260934 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:38:38.261009 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:38:38.261595 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:38.262077 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:38.262834 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:38.263049 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:38.263157 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:38.265594 (+ 2437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:38:38.265599 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:38:38.265603 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:38:38.265631 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:38:38.267185 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:38:38.267577 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:38:38.267579 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:38:38.267580 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:38:38.267667 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:38:38.267960 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:38:38.267964 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:38:38.267966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:38:38.268271 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:38:38.268572 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0925 22:38:38.268810 (+ 238us) server_negotiation.cc:299] Negotiation successful
0925 22:38:38.269426 (+ 616us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 22:38:38.272678 20691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8aea6bd8085c4ff6998e58e4157d8ef2: Bootstrap starting.
I20260925 22:38:38.273232 7131 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:38:38.276399 20691 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8aea6bd8085c4ff6998e58e4157d8ef2: Neither blocks nor log segments found. Creating new log.
I20260925 22:38:38.277518 20691 log.cc:824] T 00000000000000000000000000000000 P 8aea6bd8085c4ff6998e58e4157d8ef2: Log is configured to *not* fsync() on all Append() calls
I20260925 22:38:38.282694 20691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8aea6bd8085c4ff6998e58e4157d8ef2: No bootstrap required, opened a new log
I20260925 22:38:38.285880 20618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:38.275912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40194 (local address 127.6.246.253:39475)
0925 22:38:38.276053 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 22:38:38.276058 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:38:38.276703 (+ 645us) server_negotiation.cc:407] Connection header received
0925 22:38:38.276800 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:38:38.276805 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:38:38.276864 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:38:38.276950 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:38:38.278234 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:38.279036 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:38.279973 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:38.280322 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:38.280419 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:38.282157 (+ 1738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:38:38.282163 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:38:38.282167 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:38:38.282198 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:38:38.283882 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:38:38.284558 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:38:38.284562 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:38:38.284564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:38:38.284656 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:38:38.284966 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:38:38.284969 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:38:38.284972 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:38:38.285189 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:38:38.285332 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0925 22:38:38.285556 (+ 224us) server_negotiation.cc:299] Negotiation successful
0925 22:38:38.285663 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 22:38:38.287303 20545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:38.276024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51046 (local address 127.6.246.254:40285)
0925 22:38:38.276162 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 22:38:38.276166 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:38:38.276574 (+ 408us) server_negotiation.cc:407] Connection header received
0925 22:38:38.276808 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:38:38.276812 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:38:38.276875 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:38:38.276942 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:38:38.277934 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:38.278510 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:38.279563 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:38.279919 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:38.280081 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:38.281942 (+ 1861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:38:38.281946 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:38:38.281948 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:38:38.281976 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:38:38.283764 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:38:38.284492 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:38:38.284499 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:38:38.284504 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:38:38.284599 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:38:38.284883 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:38:38.284887 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:38:38.284889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:38:38.285184 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:38:38.285347 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 22:38:38.286999 (+ 1652us) server_negotiation.cc:299] Negotiation successful
0925 22:38:38.287138 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 22:38:38.287518 20691 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8aea6bd8085c4ff6998e58e4157d8ef2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68d3e3151a514d7b96f9de33facbfca0" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 40285 } } peers { permanent_uuid: "774b9fca766248bab60b9732783ec785" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 39475 } } peers { permanent_uuid: "8aea6bd8085c4ff6998e58e4157d8ef2" member_type: VOTER last_known_addr { host: "127.6.246.252" port: 39919 } }
I20260925 22:38:38.287834 20691 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8aea6bd8085c4ff6998e58e4157d8ef2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:38:38.287910 20691 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8aea6bd8085c4ff6998e58e4157d8ef2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8aea6bd8085c4ff6998e58e4157d8ef2, State: Initialized, Role: FOLLOWER
I20260925 22:38:38.288579 20691 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8aea6bd8085c4ff6998e58e4157d8ef2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68d3e3151a514d7b96f9de33facbfca0" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 40285 } } peers { permanent_uuid: "774b9fca766248bab60b9732783ec785" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 39475 } } peers { permanent_uuid: "8aea6bd8085c4ff6998e58e4157d8ef2" member_type: VOTER last_known_addr { host: "127.6.246.252" port: 39919 } }
I20260925 22:38:38.289170 20704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8aea6bd8085c4ff6998e58e4157d8ef2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68d3e3151a514d7b96f9de33facbfca0" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 40285 } } peers { permanent_uuid: "774b9fca766248bab60b9732783ec785" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 39475 } } peers { permanent_uuid: "8aea6bd8085c4ff6998e58e4157d8ef2" member_type: VOTER last_known_addr { host: "127.6.246.252" port: 39919 } } }
I20260925 22:38:38.289330 20704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8aea6bd8085c4ff6998e58e4157d8ef2 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:38:38.290246 20691 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8aea6bd8085c4ff6998e58e4157d8ef2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:38:38.294167 20694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:38.276328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39138 (local address 127.6.246.252:39919)
0925 22:38:38.276489 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 22:38:38.276495 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:38:38.277835 (+ 1340us) server_negotiation.cc:407] Connection header received
0925 22:38:38.277972 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:38:38.277976 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:38:38.278029 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:38:38.278101 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:38:38.278994 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:38.279682 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:38.288564 (+ 8882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:38.288824 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:38.288914 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:38.290667 (+ 1753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:38:38.290671 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:38:38.290674 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:38:38.290705 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:38:38.292201 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:38:38.292729 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:38:38.292733 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:38:38.292734 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:38:38.292816 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:38:38.293123 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:38:38.293126 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:38:38.293128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:38:38.293391 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:38:38.293534 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0925 22:38:38.293858 (+ 324us) server_negotiation.cc:299] Negotiation successful
0925 22:38:38.293994 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260925 22:38:38.297591 20715 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8aea6bd8085c4ff6998e58e4157d8ef2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:38:38.297669 20715 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:38:38.294761 20623 master_runner.cc:186] Error getting master registration for 127.6.246.254:40285: OK, instance_id { permanent_uuid: "68d3e3151a514d7b96f9de33facbfca0" instance_seqno: 1790375917708558 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 22:38:38.298084 20623 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 22:38:38.353641 20616 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.246.252" port: 39919 } attempt: 4
I20260925 22:38:38.356233 20541 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.246.252" port: 39919 } attempt: 4
Sep 25 22:38:38 dist-test-slave-wjdr krb5kdc[20460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375917, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.253@KRBTEST.COM for kudu/127.6.246.252@KRBTEST.COM
Sep 25 22:38:38 dist-test-slave-wjdr krb5kdc[20460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375917, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.254@KRBTEST.COM for kudu/127.6.246.252@KRBTEST.COM
I20260925 22:38:38.363714 20620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:38.354163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.252:39919 (local address 127.0.0.1:39146)
0925 22:38:38.354330 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:38.354346 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:38:38.354437 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:38:38.354683 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:38:38.354686 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:38:38.354938 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:38:38.355077 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:38:38.355090 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:38:38.356019 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:38:38.356023 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:38:38.356956 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:38:38.356967 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:38:38.357687 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:38:38.357690 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:38:38.357791 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:38.358377 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:38:38.358393 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:38:38.359994 (+ 1601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:38:38.361927 (+ 1933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:38:38.361931 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:38:38.361933 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:38:38.362200 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:38:38.362502 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:38:38.362505 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:38:38.362507 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:38:38.362595 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:38:38.363025 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:38:38.363031 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:38:38.363162 (+ 131us) client_negotiation.cc:769] Sending connection context
0925 22:38:38.363301 (+ 139us) client_negotiation.cc:240] Negotiation successful
0925 22:38:38.363474 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260925 22:38:38.364567 20694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:38.354262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39146 (local address 127.6.246.252:39919)
0925 22:38:38.354419 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 22:38:38.354423 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:38:38.354436 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:38:38.354532 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:38:38.354534 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:38:38.354574 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:38:38.354640 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:38:38.355228 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:38.355895 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:38.357100 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:38.357466 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:38.357559 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:38.360098 (+ 2539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:38:38.360102 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:38:38.360105 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:38:38.360123 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:38:38.361824 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:38:38.362296 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:38:38.362300 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:38:38.362301 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:38:38.362389 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:38:38.362692 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:38:38.362695 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:38:38.362697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:38:38.362908 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:38:38.363007 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 22:38:38.364229 (+ 1222us) server_negotiation.cc:299] Negotiation successful
0925 22:38:38.364356 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":44}
I20260925 22:38:38.366441 20544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:38.356732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.252:39919 (local address 127.0.0.1:39152)
0925 22:38:38.357005 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:38.357023 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 22:38:38.357109 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:38:38.357412 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:38:38.357415 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:38:38.357620 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:38:38.357717 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:38:38.357722 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:38:38.358650 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:38:38.358655 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:38:38.359336 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:38:38.359351 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:38:38.359971 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:38:38.359974 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:38:38.360076 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:38.360564 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:38:38.360582 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:38:38.361973 (+ 1391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:38:38.364909 (+ 2936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:38:38.364913 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:38:38.364915 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:38:38.365169 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:38:38.365481 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:38:38.365485 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:38:38.365486 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:38:38.365568 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:38:38.365911 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:38:38.365914 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:38:38.366018 (+ 104us) client_negotiation.cc:769] Sending connection context
0925 22:38:38.366122 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 22:38:38.366262 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175}
I20260925 22:38:38.366451 20717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:38.356775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39152 (local address 127.6.246.252:39919)
0925 22:38:38.357178 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0925 22:38:38.357190 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 22:38:38.357203 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:38:38.357241 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:38:38.357243 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:38:38.357287 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:38:38.357395 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:38:38.357824 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:38.358543 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:38.359482 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:38.359847 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:38.359950 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:38.362140 (+ 2190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:38:38.362144 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:38:38.362148 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:38:38.362177 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:38:38.364766 (+ 2589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:38:38.365288 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:38:38.365291 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:38:38.365292 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:38:38.365370 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:38:38.365658 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:38:38.365660 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:38:38.365661 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:38:38.365805 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:38:38.365920 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 22:38:38.366171 (+ 251us) server_negotiation.cc:299] Negotiation successful
0925 22:38:38.366272 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":325,"thread_start_us":141,"threads_started":1}
I20260925 22:38:38.368634 20616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 774b9fca766248bab60b9732783ec785: Bootstrap starting.
I20260925 22:38:38.369364 20541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68d3e3151a514d7b96f9de33facbfca0: Bootstrap starting.
I20260925 22:38:38.371560 20541 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68d3e3151a514d7b96f9de33facbfca0: Neither blocks nor log segments found. Creating new log.
I20260925 22:38:38.371583 20616 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 774b9fca766248bab60b9732783ec785: Neither blocks nor log segments found. Creating new log.
I20260925 22:38:38.372334 20541 log.cc:824] T 00000000000000000000000000000000 P 68d3e3151a514d7b96f9de33facbfca0: Log is configured to *not* fsync() on all Append() calls
I20260925 22:38:38.372426 20616 log.cc:824] T 00000000000000000000000000000000 P 774b9fca766248bab60b9732783ec785: Log is configured to *not* fsync() on all Append() calls
I20260925 22:38:38.374485 20541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68d3e3151a514d7b96f9de33facbfca0: No bootstrap required, opened a new log
I20260925 22:38:38.377761 20541 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68d3e3151a514d7b96f9de33facbfca0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68d3e3151a514d7b96f9de33facbfca0" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 40285 } } peers { permanent_uuid: "774b9fca766248bab60b9732783ec785" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 39475 } } peers { permanent_uuid: "8aea6bd8085c4ff6998e58e4157d8ef2" member_type: VOTER last_known_addr { host: "127.6.246.252" port: 39919 } }
I20260925 22:38:38.378003 20541 raft_consensus.cc:374] T 00000000000000000000000000000000 P 68d3e3151a514d7b96f9de33facbfca0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:38:38.378077 20541 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68d3e3151a514d7b96f9de33facbfca0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68d3e3151a514d7b96f9de33facbfca0, State: Initialized, Role: FOLLOWER
I20260925 22:38:38.378582 20541 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68d3e3151a514d7b96f9de33facbfca0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68d3e3151a514d7b96f9de33facbfca0" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 40285 } } peers { permanent_uuid: "774b9fca766248bab60b9732783ec785" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 39475 } } peers { permanent_uuid: "8aea6bd8085c4ff6998e58e4157d8ef2" member_type: VOTER last_known_addr { host: "127.6.246.252" port: 39919 } }
I20260925 22:38:38.379056 20720 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68d3e3151a514d7b96f9de33facbfca0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68d3e3151a514d7b96f9de33facbfca0" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 40285 } } peers { permanent_uuid: "774b9fca766248bab60b9732783ec785" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 39475 } } peers { permanent_uuid: "8aea6bd8085c4ff6998e58e4157d8ef2" member_type: VOTER last_known_addr { host: "127.6.246.252" port: 39919 } } }
I20260925 22:38:38.379187 20720 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68d3e3151a514d7b96f9de33facbfca0 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:38:38.379835 20616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 774b9fca766248bab60b9732783ec785: No bootstrap required, opened a new log
I20260925 22:38:38.379944 20541 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68d3e3151a514d7b96f9de33facbfca0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:38:38.386057 20616 raft_consensus.cc:348] T 00000000000000000000000000000000 P 774b9fca766248bab60b9732783ec785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68d3e3151a514d7b96f9de33facbfca0" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 40285 } } peers { permanent_uuid: "774b9fca766248bab60b9732783ec785" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 39475 } } peers { permanent_uuid: "8aea6bd8085c4ff6998e58e4157d8ef2" member_type: VOTER last_known_addr { host: "127.6.246.252" port: 39919 } }
I20260925 22:38:38.386269 20616 raft_consensus.cc:374] T 00000000000000000000000000000000 P 774b9fca766248bab60b9732783ec785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:38:38.386329 20616 raft_consensus.cc:729] T 00000000000000000000000000000000 P 774b9fca766248bab60b9732783ec785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 774b9fca766248bab60b9732783ec785, State: Initialized, Role: FOLLOWER
W20260925 22:38:38.386679 20731 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 68d3e3151a514d7b96f9de33facbfca0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:38:38.386792 20731 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:38:38.386804 20616 consensus_queue.cc:260] T 00000000000000000000000000000000 P 774b9fca766248bab60b9732783ec785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68d3e3151a514d7b96f9de33facbfca0" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 40285 } } peers { permanent_uuid: "774b9fca766248bab60b9732783ec785" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 39475 } } peers { permanent_uuid: "8aea6bd8085c4ff6998e58e4157d8ef2" member_type: VOTER last_known_addr { host: "127.6.246.252" port: 39919 } }
I20260925 22:38:38.387276 20733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 774b9fca766248bab60b9732783ec785 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68d3e3151a514d7b96f9de33facbfca0" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 40285 } } peers { permanent_uuid: "774b9fca766248bab60b9732783ec785" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 39475 } } peers { permanent_uuid: "8aea6bd8085c4ff6998e58e4157d8ef2" member_type: VOTER last_known_addr { host: "127.6.246.252" port: 39919 } } }
I20260925 22:38:38.387466 20733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 774b9fca766248bab60b9732783ec785 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:38:38.388088 20616 sys_catalog.cc:564] T 00000000000000000000000000000000 P 774b9fca766248bab60b9732783ec785 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:38:38.384792 20473 master_runner.cc:186] Error getting master registration for 127.6.246.253:39475: OK, instance_id { permanent_uuid: "774b9fca766248bab60b9732783ec785" instance_seqno: 1790375917954203 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 22:38:38.388846 20473 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260925 22:38:38.394129 20744 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 774b9fca766248bab60b9732783ec785: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:38:38.394263 20744 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:38:38.396673 20548 master_runner.cc:263] No leader master found from master 774b9fca766248bab60b9732783ec785
I20260925 22:38:38.396735 20548 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260925 22:38:38.449940 20569 tablet.cc:2369] T 00000000000000000000000000000000 P 774b9fca766248bab60b9732783ec785: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 22:38:38.454496 20494 tablet.cc:2369] T 00000000000000000000000000000000 P 68d3e3151a514d7b96f9de33facbfca0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 22:38:38.457461 20644 tablet.cc:2369] T 00000000000000000000000000000000 P 8aea6bd8085c4ff6998e58e4157d8ef2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:38:38.459233 20733 raft_consensus.cc:482] T 00000000000000000000000000000000 P 774b9fca766248bab60b9732783ec785 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:38:38.459411 20733 raft_consensus.cc:504] T 00000000000000000000000000000000 P 774b9fca766248bab60b9732783ec785 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68d3e3151a514d7b96f9de33facbfca0" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 40285 } } peers { permanent_uuid: "774b9fca766248bab60b9732783ec785" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 39475 } } peers { permanent_uuid: "8aea6bd8085c4ff6998e58e4157d8ef2" member_type: VOTER last_known_addr { host: "127.6.246.252" port: 39919 } }
I20260925 22:38:38.460636 20733 leader_election.cc:288] T 00000000000000000000000000000000 P 774b9fca766248bab60b9732783ec785 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68d3e3151a514d7b96f9de33facbfca0 (127.6.246.254:40285), 8aea6bd8085c4ff6998e58e4157d8ef2 (127.6.246.252:39919)
I20260925 22:38:38.461440 20515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "774b9fca766248bab60b9732783ec785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68d3e3151a514d7b96f9de33facbfca0" is_pre_election: true
I20260925 22:38:38.461745 20515 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 68d3e3151a514d7b96f9de33facbfca0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 774b9fca766248bab60b9732783ec785 in term 0.
I20260925 22:38:38.462487 20665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "774b9fca766248bab60b9732783ec785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8aea6bd8085c4ff6998e58e4157d8ef2" is_pre_election: true
I20260925 22:38:38.462762 20665 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8aea6bd8085c4ff6998e58e4157d8ef2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 774b9fca766248bab60b9732783ec785 in term 0.
I20260925 22:38:38.463236 20567 leader_election.cc:302] T 00000000000000000000000000000000 P 774b9fca766248bab60b9732783ec785 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 774b9fca766248bab60b9732783ec785, 8aea6bd8085c4ff6998e58e4157d8ef2; no voters:
I20260925 22:38:38.463784 20733 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 774b9fca766248bab60b9732783ec785 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:38:38.463868 20733 raft_consensus.cc:482] T 00000000000000000000000000000000 P 774b9fca766248bab60b9732783ec785 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:38:38.463910 20733 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 774b9fca766248bab60b9732783ec785 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:38:38.465247 20733 raft_consensus.cc:504] T 00000000000000000000000000000000 P 774b9fca766248bab60b9732783ec785 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68d3e3151a514d7b96f9de33facbfca0" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 40285 } } peers { permanent_uuid: "774b9fca766248bab60b9732783ec785" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 39475 } } peers { permanent_uuid: "8aea6bd8085c4ff6998e58e4157d8ef2" member_type: VOTER last_known_addr { host: "127.6.246.252" port: 39919 } }
I20260925 22:38:38.465575 20733 leader_election.cc:288] T 00000000000000000000000000000000 P 774b9fca766248bab60b9732783ec785 [CANDIDATE]: Term 1 election: Requested vote from peers 68d3e3151a514d7b96f9de33facbfca0 (127.6.246.254:40285), 8aea6bd8085c4ff6998e58e4157d8ef2 (127.6.246.252:39919)
I20260925 22:38:38.466174 20515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "774b9fca766248bab60b9732783ec785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68d3e3151a514d7b96f9de33facbfca0"
I20260925 22:38:38.466288 20515 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68d3e3151a514d7b96f9de33facbfca0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:38:38.467538 20515 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 68d3e3151a514d7b96f9de33facbfca0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 774b9fca766248bab60b9732783ec785 in term 1.
I20260925 22:38:38.468003 20665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "774b9fca766248bab60b9732783ec785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8aea6bd8085c4ff6998e58e4157d8ef2"
I20260925 22:38:38.468122 20665 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8aea6bd8085c4ff6998e58e4157d8ef2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:38:38.469439 20665 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8aea6bd8085c4ff6998e58e4157d8ef2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 774b9fca766248bab60b9732783ec785 in term 1.
I20260925 22:38:38.469785 20567 leader_election.cc:302] T 00000000000000000000000000000000 P 774b9fca766248bab60b9732783ec785 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 774b9fca766248bab60b9732783ec785, 8aea6bd8085c4ff6998e58e4157d8ef2; no voters:
I20260925 22:38:38.472335 20733 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 774b9fca766248bab60b9732783ec785 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:38:38.472590 20733 raft_consensus.cc:686] T 00000000000000000000000000000000 P 774b9fca766248bab60b9732783ec785 [term 1 LEADER]: Becoming Leader. State: Replica: 774b9fca766248bab60b9732783ec785, State: Running, Role: LEADER
I20260925 22:38:38.472975 20733 consensus_queue.cc:237] T 00000000000000000000000000000000 P 774b9fca766248bab60b9732783ec785 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68d3e3151a514d7b96f9de33facbfca0" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 40285 } } peers { permanent_uuid: "774b9fca766248bab60b9732783ec785" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 39475 } } peers { permanent_uuid: "8aea6bd8085c4ff6998e58e4157d8ef2" member_type: VOTER last_known_addr { host: "127.6.246.252" port: 39919 } }
I20260925 22:38:38.474956 20733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 774b9fca766248bab60b9732783ec785 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 774b9fca766248bab60b9732783ec785. Latest consensus state: current_term: 1 leader_uuid: "774b9fca766248bab60b9732783ec785" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68d3e3151a514d7b96f9de33facbfca0" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 40285 } } peers { permanent_uuid: "774b9fca766248bab60b9732783ec785" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 39475 } } peers { permanent_uuid: "8aea6bd8085c4ff6998e58e4157d8ef2" member_type: VOTER last_known_addr { host: "127.6.246.252" port: 39919 } } }
I20260925 22:38:38.475117 20733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 774b9fca766248bab60b9732783ec785 [sys.catalog]: This master's current role is: LEADER
I20260925 22:38:38.475597 20748 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:38:38.476926 20748 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:38:38.482546 20515 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 68d3e3151a514d7b96f9de33facbfca0 [term 1 FOLLOWER]: Refusing update from remote peer 774b9fca766248bab60b9732783ec785: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:38:38.482561 20665 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8aea6bd8085c4ff6998e58e4157d8ef2 [term 1 FOLLOWER]: Refusing update from remote peer 774b9fca766248bab60b9732783ec785: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:38:38.483219 20746 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 774b9fca766248bab60b9732783ec785 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68d3e3151a514d7b96f9de33facbfca0" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 40285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:38:38.483464 20733 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 774b9fca766248bab60b9732783ec785 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8aea6bd8085c4ff6998e58e4157d8ef2" member_type: VOTER last_known_addr { host: "127.6.246.252" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:38:38.486919 20720 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68d3e3151a514d7b96f9de33facbfca0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 774b9fca766248bab60b9732783ec785. Latest consensus state: current_term: 1 leader_uuid: "774b9fca766248bab60b9732783ec785" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68d3e3151a514d7b96f9de33facbfca0" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 40285 } } peers { permanent_uuid: "774b9fca766248bab60b9732783ec785" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 39475 } } peers { permanent_uuid: "8aea6bd8085c4ff6998e58e4157d8ef2" member_type: VOTER last_known_addr { host: "127.6.246.252" port: 39919 } } }
I20260925 22:38:38.487035 20720 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68d3e3151a514d7b96f9de33facbfca0 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:38:38.486960 20704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8aea6bd8085c4ff6998e58e4157d8ef2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 774b9fca766248bab60b9732783ec785. Latest consensus state: current_term: 1 leader_uuid: "774b9fca766248bab60b9732783ec785" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68d3e3151a514d7b96f9de33facbfca0" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 40285 } } peers { permanent_uuid: "774b9fca766248bab60b9732783ec785" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 39475 } } peers { permanent_uuid: "8aea6bd8085c4ff6998e58e4157d8ef2" member_type: VOTER last_known_addr { host: "127.6.246.252" port: 39919 } } }
I20260925 22:38:38.487071 20704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8aea6bd8085c4ff6998e58e4157d8ef2 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:38:38.489951 20733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 774b9fca766248bab60b9732783ec785 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "774b9fca766248bab60b9732783ec785" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68d3e3151a514d7b96f9de33facbfca0" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 40285 } } peers { permanent_uuid: "774b9fca766248bab60b9732783ec785" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 39475 } } peers { permanent_uuid: "8aea6bd8085c4ff6998e58e4157d8ef2" member_type: VOTER last_known_addr { host: "127.6.246.252" port: 39919 } } }
I20260925 22:38:38.490077 20733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 774b9fca766248bab60b9732783ec785 [sys.catalog]: This master's current role is: LEADER
I20260925 22:38:38.491119 20704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8aea6bd8085c4ff6998e58e4157d8ef2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "774b9fca766248bab60b9732783ec785" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68d3e3151a514d7b96f9de33facbfca0" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 40285 } } peers { permanent_uuid: "774b9fca766248bab60b9732783ec785" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 39475 } } peers { permanent_uuid: "8aea6bd8085c4ff6998e58e4157d8ef2" member_type: VOTER last_known_addr { host: "127.6.246.252" port: 39919 } } }
I20260925 22:38:38.491228 20704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8aea6bd8085c4ff6998e58e4157d8ef2 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:38:38.491214 20751 mvcc.cc:205] Tried to move back new op lower bound from 7333379762097811456 to 7333379762068783104. Current Snapshot: MvccSnapshot[applied={T|T < 7333379762097811456}]
I20260925 22:38:38.491654 20749 mvcc.cc:205] Tried to move back new op lower bound from 7333379762097811456 to 7333379762068783104. Current Snapshot: MvccSnapshot[applied={T|T < 7333379762097811456}]
I20260925 22:38:38.494714 20748 catalog_manager.cc:1347] Generated new cluster ID: 6329c16efe97492ca7f3060748abb7d8
I20260925 22:38:38.494771 20748 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:38:38.495050 20746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 774b9fca766248bab60b9732783ec785 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "774b9fca766248bab60b9732783ec785" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68d3e3151a514d7b96f9de33facbfca0" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 40285 } } peers { permanent_uuid: "774b9fca766248bab60b9732783ec785" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 39475 } } peers { permanent_uuid: "8aea6bd8085c4ff6998e58e4157d8ef2" member_type: VOTER last_known_addr { host: "127.6.246.252" port: 39919 } } }
I20260925 22:38:38.495150 20746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 774b9fca766248bab60b9732783ec785 [sys.catalog]: This master's current role is: LEADER
I20260925 22:38:38.496160 20720 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68d3e3151a514d7b96f9de33facbfca0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "774b9fca766248bab60b9732783ec785" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68d3e3151a514d7b96f9de33facbfca0" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 40285 } } peers { permanent_uuid: "774b9fca766248bab60b9732783ec785" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 39475 } } peers { permanent_uuid: "8aea6bd8085c4ff6998e58e4157d8ef2" member_type: VOTER last_known_addr { host: "127.6.246.252" port: 39919 } } }
I20260925 22:38:38.496264 20720 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68d3e3151a514d7b96f9de33facbfca0 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:38:38.496735 20750 mvcc.cc:205] Tried to move back new op lower bound from 7333379762097811456 to 7333379762068783104. Current Snapshot: MvccSnapshot[applied={T|T < 7333379762097811456}]
I20260925 22:38:38.507792 20748 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:38:38.509092 20748 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:38:38.523821 20748 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 774b9fca766248bab60b9732783ec785: Generated new TSK 0
I20260925 22:38:38.524633 20748 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:38:39.299058 20715 catalog_manager.cc:1259] Loaded cluster ID: 6329c16efe97492ca7f3060748abb7d8
I20260925 22:38:39.299151 20715 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8aea6bd8085c4ff6998e58e4157d8ef2: loading cluster ID for follower catalog manager: success
I20260925 22:38:39.300896 20715 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8aea6bd8085c4ff6998e58e4157d8ef2: acquiring CA information for follower catalog manager: success
I20260925 22:38:39.302006 20715 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8aea6bd8085c4ff6998e58e4157d8ef2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 22:38:39.388667 20731 catalog_manager.cc:1259] Loaded cluster ID: 6329c16efe97492ca7f3060748abb7d8
I20260925 22:38:39.388795 20731 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 68d3e3151a514d7b96f9de33facbfca0: loading cluster ID for follower catalog manager: success
I20260925 22:38:39.391151 20731 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 68d3e3151a514d7b96f9de33facbfca0: acquiring CA information for follower catalog manager: success
I20260925 22:38:39.392756 20731 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 68d3e3151a514d7b96f9de33facbfca0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 22:38:39.594941 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 20473
I20260925 22:38:39.602697 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 20548
I20260925 22:38:39.609743 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 20623
2026-09-25T22:38:39Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3285 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790375752797504-7131-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:38:39 dist-test-slave-wjdr krb5kdc[20762](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:38:39 dist-test-slave-wjdr krb5kdc[20762](info): set up 2 sockets
Sep 25 22:38:39 dist-test-slave-wjdr krb5kdc[20762](info): commencing operation
krb5kdc: starting...
W20260925 22:38:40.653116 7131 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.007s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Sep 25 22:38:40 dist-test-slave-wjdr krb5kdc[20762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375920, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:38:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:38:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.246.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab.
Entry for principal kudu/127.6.246.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab.
I20260925 22:38:40.788645 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790375752797504-7131-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790375752797504-7131-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790375752797504-7131-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:34501
--webserver_interface=127.6.246.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab
--principal=kudu/127.6.246.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:38627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.246.254:34501
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:38:40.918084 20775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:38:40.918474 20775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:38:40.918552 20775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:38:40.922899 20775 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:38:40.923007 20775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:38:40.923059 20775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:38:40.923116 20775 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:38:40.923151 20775 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:38:40.929520 20775 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:38627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790375752797504-7131-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790375752797504-7131-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.246.254:34501
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab
--principal=kudu/127.6.246.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:34501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.246.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20775
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790375752797504-7131-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:38:40.930953 20775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:38:40.932232 20775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:38:40.941951 20783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:38:40.942098 20775 server_base.cc:1034] running on GCE node
W20260925 22:38:40.941951 20781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:38:40.942317 20780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:38:40.942936 20775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:38:40.944257 20775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:38:40.945423 20775 hybrid_clock.cc:648] HybridClock initialized: now 1790375920945413 us; error 58 us; skew 500 ppm
Sep 25 22:38:40 dist-test-slave-wjdr krb5kdc[20762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375920, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:38:40.948720 20775 init.cc:377] Logged in from keytab as kudu/127.6.246.254@KRBTEST.COM (short username kudu)
I20260925 22:38:40.949838 20775 webserver.cc:460] Webserver started at http://127.6.246.254:36323/ using document root <none> and password file <none>
I20260925 22:38:40.950557 20775 fs_manager.cc:359] Metadata directory not provided
I20260925 22:38:40.950624 20775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:38:40.950905 20775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:38:40.953558 20775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790375752797504-7131-0/minicluster-data/master-0/data/instance:
uuid: "5518b539be52447aaa8b395cc7f566b7"
format_stamp: "Formatted at 2026-09-25 22:38:40 on dist-test-slave-wjdr"
server_key: "20b043b1e2c75bd2fa46b99ac22389c1"
server_key_iv: "39dbbd5ef731a47734e1df42aa07580b"
server_key_version: "encryptionkey@0"
I20260925 22:38:40.954253 20775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790375752797504-7131-0/minicluster-data/master-0/wal/instance:
uuid: "5518b539be52447aaa8b395cc7f566b7"
format_stamp: "Formatted at 2026-09-25 22:38:40 on dist-test-slave-wjdr"
server_key: "20b043b1e2c75bd2fa46b99ac22389c1"
server_key_iv: "39dbbd5ef731a47734e1df42aa07580b"
server_key_version: "encryptionkey@0"
I20260925 22:38:40.958787 20775 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:38:40.961649 20790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:38:40.963117 20775 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:38:40.963263 20775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790375752797504-7131-0/minicluster-data/master-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790375752797504-7131-0/minicluster-data/master-0/wal
uuid: "5518b539be52447aaa8b395cc7f566b7"
format_stamp: "Formatted at 2026-09-25 22:38:40 on dist-test-slave-wjdr"
server_key: "20b043b1e2c75bd2fa46b99ac22389c1"
server_key_iv: "39dbbd5ef731a47734e1df42aa07580b"
server_key_version: "encryptionkey@0"
I20260925 22:38:40.963376 20775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790375752797504-7131-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790375752797504-7131-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790375752797504-7131-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:38:40.978053 20775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:38:40.981911 20775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:38:40.982151 20775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:38:40.990883 20775 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.254:34501
I20260925 22:38:40.990901 20842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.254:34501 every 8 connection(s)
I20260925 22:38:40.992094 20775 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
I20260925 22:38:40.995874 20843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:38:40.996830 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 20775
I20260925 22:38:40.996989 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790375752797504-7131-0/minicluster-data/master-0/wal/instance
I20260925 22:38:40.997352 7131 external_mini_cluster.cc:1428] Setting key 0a9a699bc8ed71f8d06c93b0e809a3eb
I20260925 22:38:41.004580 20843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5518b539be52447aaa8b395cc7f566b7: Bootstrap starting.
I20260925 22:38:41.007882 20843 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5518b539be52447aaa8b395cc7f566b7: Neither blocks nor log segments found. Creating new log.
Sep 25 22:38:41 dist-test-slave-wjdr krb5kdc[20762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375920, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
I20260925 22:38:41.009059 20843 log.cc:824] T 00000000000000000000000000000000 P 5518b539be52447aaa8b395cc7f566b7: Log is configured to *not* fsync() on all Append() calls
I20260925 22:38:41.011547 20843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5518b539be52447aaa8b395cc7f566b7: No bootstrap required, opened a new log
I20260925 22:38:41.014592 20846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:40.999072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36318 (local address 127.6.246.254:34501)
0925 22:38:40.999607 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0925 22:38:40.999631 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0925 22:38:40.999688 (+ 57us) server_negotiation.cc:407] Connection header received
0925 22:38:41.000674 (+ 986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:38:41.000704 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:38:41.001234 (+ 530us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:38:41.001679 (+ 445us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:38:41.002458 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:41.004046 (+ 1588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:41.005050 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:41.005780 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:41.005943 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:41.009087 (+ 3144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:38:41.009106 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:38:41.009122 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:38:41.009171 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:38:41.011188 (+ 2017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:38:41.011710 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:38:41.011719 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:38:41.011726 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:38:41.011856 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:38:41.012174 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:38:41.012176 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:38:41.012178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:38:41.012760 (+ 582us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:38:41.012928 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 22:38:41.013272 (+ 344us) server_negotiation.cc:299] Negotiation successful
0925 22:38:41.013621 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":138,"threads_started":1}
I20260925 22:38:41.016520 20843 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5518b539be52447aaa8b395cc7f566b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5518b539be52447aaa8b395cc7f566b7" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 34501 } }
I20260925 22:38:41.016845 20843 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5518b539be52447aaa8b395cc7f566b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:38:41.016938 20843 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5518b539be52447aaa8b395cc7f566b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5518b539be52447aaa8b395cc7f566b7, State: Initialized, Role: FOLLOWER
I20260925 22:38:41.017611 20843 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5518b539be52447aaa8b395cc7f566b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5518b539be52447aaa8b395cc7f566b7" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 34501 } }
I20260925 22:38:41.017797 20843 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5518b539be52447aaa8b395cc7f566b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:38:41.017884 20843 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5518b539be52447aaa8b395cc7f566b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:38:41.017998 20843 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5518b539be52447aaa8b395cc7f566b7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:38:41.018143 7131 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:38:41.019526 20843 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5518b539be52447aaa8b395cc7f566b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5518b539be52447aaa8b395cc7f566b7" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 34501 } }
I20260925 22:38:41.019917 20843 leader_election.cc:302] T 00000000000000000000000000000000 P 5518b539be52447aaa8b395cc7f566b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5518b539be52447aaa8b395cc7f566b7; no voters:
I20260925 22:38:41.020334 20843 leader_election.cc:288] T 00000000000000000000000000000000 P 5518b539be52447aaa8b395cc7f566b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:38:41.020478 20848 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5518b539be52447aaa8b395cc7f566b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:38:41.020747 20848 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5518b539be52447aaa8b395cc7f566b7 [term 1 LEADER]: Becoming Leader. State: Replica: 5518b539be52447aaa8b395cc7f566b7, State: Running, Role: LEADER
I20260925 22:38:41.021138 20848 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5518b539be52447aaa8b395cc7f566b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5518b539be52447aaa8b395cc7f566b7" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 34501 } }
I20260925 22:38:41.021873 20843 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5518b539be52447aaa8b395cc7f566b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:38:41.023182 20849 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5518b539be52447aaa8b395cc7f566b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5518b539be52447aaa8b395cc7f566b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5518b539be52447aaa8b395cc7f566b7" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 34501 } } }
I20260925 22:38:41.023332 20849 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5518b539be52447aaa8b395cc7f566b7 [sys.catalog]: This master's current role is: LEADER
I20260925 22:38:41.023660 20850 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5518b539be52447aaa8b395cc7f566b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5518b539be52447aaa8b395cc7f566b7. Latest consensus state: current_term: 1 leader_uuid: "5518b539be52447aaa8b395cc7f566b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5518b539be52447aaa8b395cc7f566b7" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 34501 } } }
I20260925 22:38:41.023756 20857 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:38:41.024585 20850 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5518b539be52447aaa8b395cc7f566b7 [sys.catalog]: This master's current role is: LEADER
I20260925 22:38:41.027777 20857 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:38:41.034853 20857 catalog_manager.cc:1347] Generated new cluster ID: 7c50231190224921be106d4bfe19df25
I20260925 22:38:41.034938 20857 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:38:41.042563 20857 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:38:41.043804 20857 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:38:41.051749 20857 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5518b539be52447aaa8b395cc7f566b7: Generated new TSK 0
I20260925 22:38:41.052731 20857 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:38:41.075273 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 20775
2026-09-25T22:38:41Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1463 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:38:41 dist-test-slave-wjdr krb5kdc[20870](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:38:41 dist-test-slave-wjdr krb5kdc[20870](info): set up 2 sockets
Sep 25 22:38:41 dist-test-slave-wjdr krb5kdc[20870](info): commencing operation
krb5kdc: starting...
W20260925 22:38:42.125214 7131 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Sep 25 22:38:42 dist-test-slave-wjdr krb5kdc[20870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:38:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:38:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.246.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab.
Entry for principal kudu/127.6.246.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab.
I20260925 22:38:42.259290 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:43017
--webserver_interface=127.6.246.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab
--principal=kudu/127.6.246.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:34883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.246.254:43017
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:38:42.388743 20883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:38:42.389161 20883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:38:42.389243 20883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:38:42.393728 20883 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:38:42.393882 20883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:38:42.393920 20883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:38:42.393959 20883 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:38:42.393997 20883 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:38:42.399837 20883 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:34883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.246.254:43017
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab
--principal=kudu/127.6.246.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:43017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.246.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20883
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:38:42.401285 20883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:38:42.402593 20883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:38:42.412997 20889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:38:42.412955 20891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:38:42.413076 20883 server_base.cc:1034] running on GCE node
W20260925 22:38:42.412955 20888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:38:42.414059 20883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:38:42.415700 20883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:38:42.416910 20883 hybrid_clock.cc:648] HybridClock initialized: now 1790375922416880 us; error 56 us; skew 500 ppm
Sep 25 22:38:42 dist-test-slave-wjdr krb5kdc[20870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375922, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:38:42.420364 20883 init.cc:377] Logged in from keytab as kudu/127.6.246.254@KRBTEST.COM (short username kudu)
I20260925 22:38:42.421627 20883 webserver.cc:460] Webserver started at http://127.6.246.254:33505/ using document root <none> and password file <none>
I20260925 22:38:42.422360 20883 fs_manager.cc:359] Metadata directory not provided
I20260925 22:38:42.422459 20883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:38:42.422735 20883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:38:42.425489 20883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/master-0/data/instance:
uuid: "761938f0951e48a0910db98fde88b05f"
format_stamp: "Formatted at 2026-09-25 22:38:42 on dist-test-slave-wjdr"
server_key: "18768535b2555fb849ed97cf45aebc3c"
server_key_iv: "5804c09b287daae23dfb0a1be6d786f4"
server_key_version: "encryptionkey@0"
I20260925 22:38:42.426230 20883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/master-0/wal/instance:
uuid: "761938f0951e48a0910db98fde88b05f"
format_stamp: "Formatted at 2026-09-25 22:38:42 on dist-test-slave-wjdr"
server_key: "18768535b2555fb849ed97cf45aebc3c"
server_key_iv: "5804c09b287daae23dfb0a1be6d786f4"
server_key_version: "encryptionkey@0"
I20260925 22:38:42.431167 20883 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 22:38:42.434486 20898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:38:42.436141 20883 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:38:42.436328 20883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/master-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/master-0/wal
uuid: "761938f0951e48a0910db98fde88b05f"
format_stamp: "Formatted at 2026-09-25 22:38:42 on dist-test-slave-wjdr"
server_key: "18768535b2555fb849ed97cf45aebc3c"
server_key_iv: "5804c09b287daae23dfb0a1be6d786f4"
server_key_version: "encryptionkey@0"
I20260925 22:38:42.436514 20883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:38:42.459286 20883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:38:42.463692 20883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:38:42.463981 20883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:38:42.473148 20883 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.254:43017
I20260925 22:38:42.473498 20950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.254:43017 every 8 connection(s)
I20260925 22:38:42.474464 20883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
I20260925 22:38:42.477298 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 20883
I20260925 22:38:42.477442 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/master-0/wal/instance
I20260925 22:38:42.477854 7131 external_mini_cluster.cc:1428] Setting key 325caf1f987f759263c7bde56f849616
I20260925 22:38:42.478451 20951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:38:42.486821 20951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 761938f0951e48a0910db98fde88b05f: Bootstrap starting.
Sep 25 22:38:42 dist-test-slave-wjdr krb5kdc[20870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
I20260925 22:38:42.490284 20951 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 761938f0951e48a0910db98fde88b05f: Neither blocks nor log segments found. Creating new log.
I20260925 22:38:42.491339 20951 log.cc:824] T 00000000000000000000000000000000 P 761938f0951e48a0910db98fde88b05f: Log is configured to *not* fsync() on all Append() calls
I20260925 22:38:42.493938 20951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 761938f0951e48a0910db98fde88b05f: No bootstrap required, opened a new log
I20260925 22:38:42.494436 20954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:42.479630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56128 (local address 127.6.246.254:43017)
0925 22:38:42.480132 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0925 22:38:42.480151 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 22:38:42.480194 (+ 43us) server_negotiation.cc:407] Connection header received
0925 22:38:42.481017 (+ 823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:38:42.481040 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:38:42.481438 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:38:42.481784 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:38:42.482760 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:42.483849 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:42.485102 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:42.485624 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:42.485752 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:42.488861 (+ 3109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:38:42.488869 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:38:42.488888 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:38:42.488926 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:38:42.491050 (+ 2124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:38:42.491524 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:38:42.491531 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:38:42.491537 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:38:42.491686 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:38:42.491981 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:38:42.491984 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:38:42.491985 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:38:42.492540 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:38:42.492764 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0925 22:38:42.493102 (+ 338us) server_negotiation.cc:299] Negotiation successful
0925 22:38:42.493450 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":164,"threads_started":1}
I20260925 22:38:42.498553 20951 raft_consensus.cc:348] T 00000000000000000000000000000000 P 761938f0951e48a0910db98fde88b05f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "761938f0951e48a0910db98fde88b05f" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 43017 } }
I20260925 22:38:42.498890 20951 raft_consensus.cc:374] T 00000000000000000000000000000000 P 761938f0951e48a0910db98fde88b05f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:38:42.498968 20951 raft_consensus.cc:729] T 00000000000000000000000000000000 P 761938f0951e48a0910db98fde88b05f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 761938f0951e48a0910db98fde88b05f, State: Initialized, Role: FOLLOWER
I20260925 22:38:42.499511 20951 consensus_queue.cc:260] T 00000000000000000000000000000000 P 761938f0951e48a0910db98fde88b05f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "761938f0951e48a0910db98fde88b05f" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 43017 } }
I20260925 22:38:42.499684 20951 raft_consensus.cc:388] T 00000000000000000000000000000000 P 761938f0951e48a0910db98fde88b05f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:38:42.499775 20951 raft_consensus.cc:482] T 00000000000000000000000000000000 P 761938f0951e48a0910db98fde88b05f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:38:42.499902 20951 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 761938f0951e48a0910db98fde88b05f [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:38:42.501180 20951 raft_consensus.cc:504] T 00000000000000000000000000000000 P 761938f0951e48a0910db98fde88b05f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "761938f0951e48a0910db98fde88b05f" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 43017 } }
I20260925 22:38:42.501633 20951 leader_election.cc:302] T 00000000000000000000000000000000 P 761938f0951e48a0910db98fde88b05f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 761938f0951e48a0910db98fde88b05f; no voters:
I20260925 22:38:42.502115 20951 leader_election.cc:288] T 00000000000000000000000000000000 P 761938f0951e48a0910db98fde88b05f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:38:42.502225 20956 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 761938f0951e48a0910db98fde88b05f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:38:42.502432 20956 raft_consensus.cc:686] T 00000000000000000000000000000000 P 761938f0951e48a0910db98fde88b05f [term 1 LEADER]: Becoming Leader. State: Replica: 761938f0951e48a0910db98fde88b05f, State: Running, Role: LEADER
I20260925 22:38:42.502853 20956 consensus_queue.cc:237] T 00000000000000000000000000000000 P 761938f0951e48a0910db98fde88b05f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "761938f0951e48a0910db98fde88b05f" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 43017 } }
I20260925 22:38:42.503808 20951 sys_catalog.cc:564] T 00000000000000000000000000000000 P 761938f0951e48a0910db98fde88b05f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:38:42.504997 20957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 761938f0951e48a0910db98fde88b05f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "761938f0951e48a0910db98fde88b05f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "761938f0951e48a0910db98fde88b05f" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 43017 } } }
I20260925 22:38:42.505055 20958 sys_catalog.cc:455] T 00000000000000000000000000000000 P 761938f0951e48a0910db98fde88b05f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 761938f0951e48a0910db98fde88b05f. Latest consensus state: current_term: 1 leader_uuid: "761938f0951e48a0910db98fde88b05f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "761938f0951e48a0910db98fde88b05f" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 43017 } } }
I20260925 22:38:42.505177 20957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 761938f0951e48a0910db98fde88b05f [sys.catalog]: This master's current role is: LEADER
I20260925 22:38:42.505179 20958 sys_catalog.cc:458] T 00000000000000000000000000000000 P 761938f0951e48a0910db98fde88b05f [sys.catalog]: This master's current role is: LEADER
I20260925 22:38:42.505698 20965 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:38:42.509765 20965 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:38:42.517231 20965 catalog_manager.cc:1347] Generated new cluster ID: 79b9eef78b7641a9a88d8832afd1e153
I20260925 22:38:42.517341 20965 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:38:42.528704 20965 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:38:42.530316 20965 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:38:42.538750 20965 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 761938f0951e48a0910db98fde88b05f: Generated new TSK 0
I20260925 22:38:42.539866 20965 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.246.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.193.keytab.
Entry for principal kudu/127.6.246.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.193.keytab.
I20260925 22:38:42.581341 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.193:0
--local_ip_for_outbound_sockets=127.6.246.193
--webserver_interface=127.6.246.193
--webserver_port=0
--tserver_master_addrs=127.6.246.254:43017
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.193.keytab
--principal=kudu/127.6.246.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:34883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:38:42.714913 20977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:38:42.715341 20977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:38:42.715420 20977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:38:42.720129 20977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:38:42.720245 20977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:38:42.720347 20977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.193
I20260925 22:38:42.726246 20977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:34883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.193.keytab
--principal=kudu/127.6.246.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.246.193
--webserver_port=0
--tserver_master_addrs=127.6.246.254:43017
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.193
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:38:42.727644 20977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:38:42.728961 20977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:38:42.740482 20982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:38:42.740486 20983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:38:42.740523 20985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:38:42.740831 20977 server_base.cc:1034] running on GCE node
I20260925 22:38:42.741338 20977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:38:42.742142 20977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:38:42.743304 20977 hybrid_clock.cc:648] HybridClock initialized: now 1790375922743275 us; error 46 us; skew 500 ppm
Sep 25 22:38:42 dist-test-slave-wjdr krb5kdc[20870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375922, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:38:42.746373 20977 init.cc:377] Logged in from keytab as kudu/127.6.246.193@KRBTEST.COM (short username kudu)
I20260925 22:38:42.747326 20977 webserver.cc:460] Webserver started at http://127.6.246.193:38055/ using document root <none> and password file <none>
I20260925 22:38:42.747964 20977 fs_manager.cc:359] Metadata directory not provided
I20260925 22:38:42.748025 20977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:38:42.748224 20977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:38:42.750708 20977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-0/data/instance:
uuid: "593624e63fab46768718c022af35acde"
format_stamp: "Formatted at 2026-09-25 22:38:42 on dist-test-slave-wjdr"
server_key: "65166019af2a3076ec1047c39d91369d"
server_key_iv: "abccb242a4c3c976be25e55b993397a4"
server_key_version: "encryptionkey@0"
I20260925 22:38:42.751302 20977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-0/wal/instance:
uuid: "593624e63fab46768718c022af35acde"
format_stamp: "Formatted at 2026-09-25 22:38:42 on dist-test-slave-wjdr"
server_key: "65166019af2a3076ec1047c39d91369d"
server_key_iv: "abccb242a4c3c976be25e55b993397a4"
server_key_version: "encryptionkey@0"
I20260925 22:38:42.755829 20977 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260925 22:38:42.758908 20992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:38:42.760600 20977 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:38:42.760744 20977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-0/wal
uuid: "593624e63fab46768718c022af35acde"
format_stamp: "Formatted at 2026-09-25 22:38:42 on dist-test-slave-wjdr"
server_key: "65166019af2a3076ec1047c39d91369d"
server_key_iv: "abccb242a4c3c976be25e55b993397a4"
server_key_version: "encryptionkey@0"
I20260925 22:38:42.760872 20977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:38:42.785315 20977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:38:42.788856 20977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:38:42.789255 20977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:38:42.790257 20977 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:38:42.791327 20977 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:38:42.791381 20977 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:38:42.791457 20977 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:38:42.791488 20977 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:38:42.802353 20977 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.193:33913
I20260925 22:38:42.802376 21105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.193:33913 every 8 connection(s)
I20260925 22:38:42.803519 20977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
I20260925 22:38:42.810425 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 20977
I20260925 22:38:42.810616 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-0/wal/instance
I20260925 22:38:42.810972 7131 external_mini_cluster.cc:1428] Setting key 4f3c4a3385001a5cc63a6de9b7bb1cb7
Sep 25 22:38:42 dist-test-slave-wjdr krb5kdc[20870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375922, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.193@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
I20260925 22:38:42.820542 20954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:42.805948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.193:51559 (local address 127.6.246.254:43017)
0925 22:38:42.806149 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 22:38:42.806154 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:38:42.807528 (+ 1374us) server_negotiation.cc:407] Connection header received
0925 22:38:42.808689 (+ 1161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:38:42.808693 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:38:42.808766 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:38:42.808870 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:38:42.810299 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:42.811006 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:42.812522 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:42.812845 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:42.812979 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:42.816010 (+ 3031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:38:42.816018 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:38:42.816024 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:38:42.816067 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:38:42.817918 (+ 1851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:38:42.818486 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:38:42.818492 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:38:42.818494 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:38:42.818599 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:38:42.818965 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:38:42.818971 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:38:42.818976 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:38:42.819304 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:38:42.819432 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 22:38:42.820042 (+ 610us) server_negotiation.cc:299] Negotiation successful
0925 22:38:42.820258 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 22:38:42.821640 21108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:42.806486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43017 (local address 127.6.246.193:51559)
0925 22:38:42.807338 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:42.807388 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 22:38:42.808386 (+ 998us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:38:42.809101 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:38:42.809109 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:38:42.809540 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:38:42.810137 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:38:42.810152 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:38:42.811189 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:38:42.811192 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:38:42.812375 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:38:42.812391 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:38:42.813037 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:38:42.813042 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:38:42.813256 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:42.814028 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:38:42.814047 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:38:42.815817 (+ 1770us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:38:42.818043 (+ 2226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:38:42.818049 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:38:42.818069 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:38:42.818376 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:38:42.818697 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:38:42.818699 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:38:42.818701 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:38:42.818868 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:38:42.819414 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:38:42.819420 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:38:42.819759 (+ 339us) client_negotiation.cc:769] Sending connection context
0925 22:38:42.819953 (+ 194us) client_negotiation.cc:240] Negotiation successful
0925 22:38:42.820306 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":137,"threads_started":1}
WARNING: no policy specified for kudu/127.6.246.194@KRBTEST.COM; defaulting to no policy
I20260925 22:38:42.823290 21106 heartbeater.cc:347] Connected to a master server at 127.6.246.254:43017
I20260925 22:38:42.823582 21106 heartbeater.cc:464] Registering TS with master...
I20260925 22:38:42.824183 21106 heartbeater.cc:511] Master 127.6.246.254:43017 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.194@KRBTEST.COM" created.
I20260925 22:38:42.826227 20915 ts_manager.cc:194] Registered new tserver with Master: 593624e63fab46768718c022af35acde (127.6.246.193:33913)
I20260925 22:38:42.828161 20915 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.246.193@KRBTEST.COM'} at 127.6.246.193:51559
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.194.keytab.
Entry for principal kudu/127.6.246.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.194.keytab.
I20260925 22:38:42.843706 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.194:0
--local_ip_for_outbound_sockets=127.6.246.194
--webserver_interface=127.6.246.194
--webserver_port=0
--tserver_master_addrs=127.6.246.254:43017
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.194.keytab
--principal=kudu/127.6.246.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:34883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:38:42.977859 21111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:38:42.978235 21111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:38:42.978291 21111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:38:42.982887 21111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:38:42.982957 21111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:38:42.983023 21111 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.194
I20260925 22:38:42.988842 21111 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:34883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.194.keytab
--principal=kudu/127.6.246.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.246.194
--webserver_port=0
--tserver_master_addrs=127.6.246.254:43017
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.194
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:38:42.990247 21111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:38:42.991549 21111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:38:43.003196 21119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:38:43.003196 21117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:38:43.003341 21111 server_base.cc:1034] running on GCE node
W20260925 22:38:43.003232 21116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:38:43.004155 21111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:38:43.005051 21111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:38:43.006214 21111 hybrid_clock.cc:648] HybridClock initialized: now 1790375923006196 us; error 38 us; skew 500 ppm
Sep 25 22:38:43 dist-test-slave-wjdr krb5kdc[20870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375923, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:38:43.009626 21111 init.cc:377] Logged in from keytab as kudu/127.6.246.194@KRBTEST.COM (short username kudu)
I20260925 22:38:43.010792 21111 webserver.cc:460] Webserver started at http://127.6.246.194:36019/ using document root <none> and password file <none>
I20260925 22:38:43.011482 21111 fs_manager.cc:359] Metadata directory not provided
I20260925 22:38:43.011544 21111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:38:43.011749 21111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:38:43.014361 21111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-1/data/instance:
uuid: "6487020f6fdd432eaecee9f7c1c8beb1"
format_stamp: "Formatted at 2026-09-25 22:38:43 on dist-test-slave-wjdr"
server_key: "4642dbdfdadc8575fad1e4e9b7d083e8"
server_key_iv: "c5ee12b563ba3927f8ba289716028424"
server_key_version: "encryptionkey@0"
I20260925 22:38:43.014976 21111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-1/wal/instance:
uuid: "6487020f6fdd432eaecee9f7c1c8beb1"
format_stamp: "Formatted at 2026-09-25 22:38:43 on dist-test-slave-wjdr"
server_key: "4642dbdfdadc8575fad1e4e9b7d083e8"
server_key_iv: "c5ee12b563ba3927f8ba289716028424"
server_key_version: "encryptionkey@0"
I20260925 22:38:43.019740 21111 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260925 22:38:43.022840 21126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:38:43.024415 21111 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 22:38:43.024585 21111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-1/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-1/wal
uuid: "6487020f6fdd432eaecee9f7c1c8beb1"
format_stamp: "Formatted at 2026-09-25 22:38:43 on dist-test-slave-wjdr"
server_key: "4642dbdfdadc8575fad1e4e9b7d083e8"
server_key_iv: "c5ee12b563ba3927f8ba289716028424"
server_key_version: "encryptionkey@0"
I20260925 22:38:43.024717 21111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:38:43.053543 21111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:38:43.057173 21111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:38:43.057358 21111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:38:43.058023 21111 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:38:43.059013 21111 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:38:43.059067 21111 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:38:43.059115 21111 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:38:43.059137 21111 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:38:43.069427 21111 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.194:45893
I20260925 22:38:43.069511 21239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.194:45893 every 8 connection(s)
I20260925 22:38:43.070533 21111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
I20260925 22:38:43.072388 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 21111
I20260925 22:38:43.072582 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-1/wal/instance
I20260925 22:38:43.072992 7131 external_mini_cluster.cc:1428] Setting key 6c68f1f5f0f6af5fd0fbcec39dfaa9c2
Sep 25 22:38:43 dist-test-slave-wjdr krb5kdc[20870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375923, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.194@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.246.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.195@KRBTEST.COM" created.
I20260925 22:38:43.088248 20954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:43.072889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.194:44505 (local address 127.6.246.254:43017)
0925 22:38:43.073094 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 22:38:43.073100 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:38:43.074541 (+ 1441us) server_negotiation.cc:407] Connection header received
0925 22:38:43.075985 (+ 1444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:38:43.075988 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:38:43.076039 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:38:43.076114 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:38:43.077912 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:43.078525 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:43.080375 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:43.080707 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:43.080826 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:43.084290 (+ 3464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:38:43.084294 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:38:43.084296 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:38:43.084323 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:38:43.086054 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:38:43.086585 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:38:43.086588 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:38:43.086589 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:38:43.086673 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:38:43.086983 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:38:43.086985 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:38:43.086986 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:38:43.087192 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:38:43.087271 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 22:38:43.087835 (+ 564us) server_negotiation.cc:299] Negotiation successful
0925 22:38:43.088030 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 22:38:43.089421 21242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:43.073474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43017 (local address 127.6.246.194:44505)
0925 22:38:43.074377 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:43.074426 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 22:38:43.075713 (+ 1287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:38:43.076331 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:38:43.076344 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:38:43.077012 (+ 668us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:38:43.077759 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:38:43.077774 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:38:43.078666 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:38:43.078673 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:38:43.080248 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:38:43.080262 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:38:43.080902 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:38:43.080909 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:38:43.081159 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:43.081909 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:38:43.081942 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:38:43.084078 (+ 2136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:38:43.086177 (+ 2099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:38:43.086183 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:38:43.086199 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:38:43.086484 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:38:43.086759 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:38:43.086762 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:38:43.086763 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:38:43.086907 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:38:43.087283 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:38:43.087288 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:38:43.087585 (+ 297us) client_negotiation.cc:769] Sending connection context
0925 22:38:43.087778 (+ 193us) client_negotiation.cc:240] Negotiation successful
0925 22:38:43.088187 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"thread_start_us":185,"threads_started":1}
I20260925 22:38:43.090965 21240 heartbeater.cc:347] Connected to a master server at 127.6.246.254:43017
I20260925 22:38:43.091300 21240 heartbeater.cc:464] Registering TS with master...
I20260925 22:38:43.091938 21240 heartbeater.cc:511] Master 127.6.246.254:43017 requested a full tablet report, sending...
I20260925 22:38:43.093392 20915 ts_manager.cc:194] Registered new tserver with Master: 6487020f6fdd432eaecee9f7c1c8beb1 (127.6.246.194:45893)
I20260925 22:38:43.094419 20915 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.246.194@KRBTEST.COM'} at 127.6.246.194:44505
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.195.keytab.
Entry for principal kudu/127.6.246.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.195.keytab.
I20260925 22:38:43.107931 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.195:0
--local_ip_for_outbound_sockets=127.6.246.195
--webserver_interface=127.6.246.195
--webserver_port=0
--tserver_master_addrs=127.6.246.254:43017
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.195.keytab
--principal=kudu/127.6.246.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:34883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:38:43.240801 21245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:38:43.241232 21245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:38:43.241291 21245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:38:43.246378 21245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:38:43.246465 21245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:38:43.246538 21245 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.195
I20260925 22:38:43.252568 21245 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:34883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.195.keytab
--principal=kudu/127.6.246.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.246.195
--webserver_port=0
--tserver_master_addrs=127.6.246.254:43017
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21245
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.195
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:38:43.253998 21245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:38:43.255450 21245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:38:43.267421 21250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:38:43.267504 21253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:38:43.267413 21251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:38:43.267831 21245 server_base.cc:1034] running on GCE node
I20260925 22:38:43.268337 21245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:38:43.269136 21245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:38:43.270334 21245 hybrid_clock.cc:648] HybridClock initialized: now 1790375923270316 us; error 40 us; skew 500 ppm
Sep 25 22:38:43 dist-test-slave-wjdr krb5kdc[20870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375923, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:38:43.287691 21245 init.cc:377] Logged in from keytab as kudu/127.6.246.195@KRBTEST.COM (short username kudu)
I20260925 22:38:43.289052 21245 webserver.cc:460] Webserver started at http://127.6.246.195:45167/ using document root <none> and password file <none>
I20260925 22:38:43.289789 21245 fs_manager.cc:359] Metadata directory not provided
I20260925 22:38:43.289883 21245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:38:43.290154 21245 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:38:43.292905 21245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-2/data/instance:
uuid: "2f130a10f1e64d5e8439fa19988dfee5"
format_stamp: "Formatted at 2026-09-25 22:38:43 on dist-test-slave-wjdr"
server_key: "07c03525d4e9c4e6ce3969574621eb3c"
server_key_iv: "2235cc6693edd0c629bd98ee65b2c830"
server_key_version: "encryptionkey@0"
I20260925 22:38:43.293594 21245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-2/wal/instance:
uuid: "2f130a10f1e64d5e8439fa19988dfee5"
format_stamp: "Formatted at 2026-09-25 22:38:43 on dist-test-slave-wjdr"
server_key: "07c03525d4e9c4e6ce3969574621eb3c"
server_key_iv: "2235cc6693edd0c629bd98ee65b2c830"
server_key_version: "encryptionkey@0"
I20260925 22:38:43.298130 21245 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:38:43.301076 21260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:38:43.302574 21245 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:38:43.302719 21245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-2/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-2/wal
uuid: "2f130a10f1e64d5e8439fa19988dfee5"
format_stamp: "Formatted at 2026-09-25 22:38:43 on dist-test-slave-wjdr"
server_key: "07c03525d4e9c4e6ce3969574621eb3c"
server_key_iv: "2235cc6693edd0c629bd98ee65b2c830"
server_key_version: "encryptionkey@0"
I20260925 22:38:43.302850 21245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:38:43.315304 21245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:38:43.318890 21245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:38:43.319123 21245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:38:43.319856 21245 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:38:43.320946 21245 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:38:43.321017 21245 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:38:43.321091 21245 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:38:43.321138 21245 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:38:43.332660 21245 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.195:43353
I20260925 22:38:43.332683 21373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.195:43353 every 8 connection(s)
I20260925 22:38:43.333951 21245 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
I20260925 22:38:43.336544 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 21245
I20260925 22:38:43.336730 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-2/wal/instance
I20260925 22:38:43.337133 7131 external_mini_cluster.cc:1428] Setting key 2dea1f0ffec3eecce413437d6c0bc116
Sep 25 22:38:43 dist-test-slave-wjdr krb5kdc[20870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375923, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.195@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
I20260925 22:38:43.350150 20954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:43.336243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.195:48627 (local address 127.6.246.254:43017)
0925 22:38:43.336428 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 22:38:43.336432 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:38:43.338452 (+ 2020us) server_negotiation.cc:407] Connection header received
0925 22:38:43.339372 (+ 920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:38:43.339375 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:38:43.339439 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:38:43.339535 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:38:43.340993 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:43.341658 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:43.342931 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:43.343188 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:43.343311 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:43.346212 (+ 2901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:38:43.346215 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:38:43.346218 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:38:43.346247 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:38:43.347868 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:38:43.348410 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:38:43.348413 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:38:43.348415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:38:43.348550 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:38:43.348917 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:38:43.348920 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:38:43.348922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:38:43.349136 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:38:43.349248 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 22:38:43.349808 (+ 560us) server_negotiation.cc:299] Negotiation successful
0925 22:38:43.349958 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 22:38:43.351384 21376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:43.336899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43017 (local address 127.6.246.195:48627)
0925 22:38:43.338106 (+ 1207us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:43.338154 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 22:38:43.339143 (+ 989us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:38:43.339741 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:38:43.339755 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:38:43.340175 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:38:43.340847 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:38:43.340862 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:38:43.341799 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:38:43.341802 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:38:43.342781 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:38:43.342790 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:38:43.343371 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:38:43.343376 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:38:43.343566 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:43.344289 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:38:43.344307 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:38:43.346030 (+ 1723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:38:43.347989 (+ 1959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:38:43.347995 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:38:43.348018 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:38:43.348296 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:38:43.348648 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:38:43.348652 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:38:43.348654 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:38:43.348813 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:38:43.349243 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:38:43.349250 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:38:43.349549 (+ 299us) client_negotiation.cc:769] Sending connection context
0925 22:38:43.349759 (+ 210us) client_negotiation.cc:240] Negotiation successful
0925 22:38:43.350149 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":942,"thread_start_us":172,"threads_started":1}
I20260925 22:38:43.352651 21374 heartbeater.cc:347] Connected to a master server at 127.6.246.254:43017
I20260925 22:38:43.352952 21374 heartbeater.cc:464] Registering TS with master...
I20260925 22:38:43.353538 21374 heartbeater.cc:511] Master 127.6.246.254:43017 requested a full tablet report, sending...
I20260925 22:38:43.354712 20915 ts_manager.cc:194] Registered new tserver with Master: 2f130a10f1e64d5e8439fa19988dfee5 (127.6.246.195:43353)
I20260925 22:38:43.355381 20915 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.246.195@KRBTEST.COM'} at 127.6.246.195:48627
I20260925 22:38:43.362943 7131 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:38:43.372996 20954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:43.365282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56144 (local address 127.6.246.254:43017)
0925 22:38:43.365414 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 22:38:43.365418 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:38:43.365674 (+ 256us) server_negotiation.cc:407] Connection header received
0925 22:38:43.365792 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:38:43.365794 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:38:43.365836 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:38:43.365942 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:38:43.366738 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:43.367290 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:43.368148 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:43.368377 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:43.368515 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:43.369947 (+ 1432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:38:43.369951 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:38:43.369954 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:38:43.369983 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:38:43.371200 (+ 1217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:38:43.371647 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:38:43.371652 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:38:43.371654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:38:43.371732 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:38:43.372002 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:38:43.372007 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:38:43.372009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:38:43.372269 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:38:43.372417 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 22:38:43.372676 (+ 259us) server_negotiation.cc:299] Negotiation successful
0925 22:38:43.372797 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 22:38:43.376076 20915 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56144:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:38:43.379781 20915 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:38:43.395851 21388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:43.390534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.193:33913 (local address 127.0.0.1:49904)
0925 22:38:43.391076 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:43.391090 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:38:43.391173 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:38:43.391931 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:38:43.391934 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:38:43.391956 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:38:43.392029 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:38:43.392033 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:38:43.393408 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:38:43.393415 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:38:43.394320 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:38:43.394330 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:38:43.395250 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:38:43.395254 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:38:43.395370 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:43.395417 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 22:38:43.395535 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 22:38:43.395667 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":106,"threads_started":1}
I20260925 22:38:43.395861 21390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:43.391025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49904 (local address 127.6.246.193:33913)
0925 22:38:43.391471 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0925 22:38:43.391477 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:38:43.391499 (+ 22us) server_negotiation.cc:407] Connection header received
0925 22:38:43.391581 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:38:43.391586 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:38:43.391755 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:38:43.391923 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:38:43.392139 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:43.393299 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:43.394451 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:43.395111 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:43.395219 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:43.395411 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0925 22:38:43.395581 (+ 170us) server_negotiation.cc:299] Negotiation successful
0925 22:38:43.395691 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":112,"threads_started":1}
I20260925 22:38:43.396605 21385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:43.389660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.195:43353 (local address 127.0.0.1:55248)
0925 22:38:43.390062 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:43.390115 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 22:38:43.390253 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:38:43.391243 (+ 990us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:38:43.391250 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:38:43.391274 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:38:43.391452 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:38:43.391459 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:38:43.393245 (+ 1786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:38:43.393249 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:38:43.394978 (+ 1729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:38:43.394985 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:38:43.396104 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:38:43.396108 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:38:43.396205 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:43.396229 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 22:38:43.396326 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 22:38:43.396419 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":227,"thread_start_us":121,"threads_started":1}
I20260925 22:38:43.396629 21386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:43.389836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55248 (local address 127.6.246.195:43353)
0925 22:38:43.390641 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0925 22:38:43.390658 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 22:38:43.390685 (+ 27us) server_negotiation.cc:407] Connection header received
0925 22:38:43.390807 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:38:43.390814 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:38:43.391043 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:38:43.391618 (+ 575us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:38:43.391798 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:43.393092 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:43.395129 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:43.395990 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:43.396116 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:43.396270 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 22:38:43.396350 (+ 80us) server_negotiation.cc:299] Negotiation successful
0925 22:38:43.396439 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":662,"thread_start_us":166,"threads_started":1}
I20260925 22:38:43.397398 21387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:43.390122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.194:45893 (local address 127.0.0.1:59156)
0925 22:38:43.390727 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:43.390742 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:38:43.390879 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:38:43.391625 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:38:43.391628 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:38:43.391644 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:38:43.391775 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:38:43.391785 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:38:43.393093 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:38:43.393100 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:38:43.394974 (+ 1874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:38:43.394985 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:38:43.396966 (+ 1981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:38:43.396969 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:38:43.397083 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:43.397107 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 22:38:43.397185 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 22:38:43.397265 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"mutex_wait_us":208,"thread_start_us":122,"threads_started":1}
I20260925 22:38:43.397480 21389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:43.390682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59156 (local address 127.6.246.194:45893)
0925 22:38:43.391222 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0925 22:38:43.391236 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 22:38:43.391273 (+ 37us) server_negotiation.cc:407] Connection header received
0925 22:38:43.391346 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:38:43.391351 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:38:43.391493 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:38:43.391650 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:38:43.391905 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:43.392937 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:43.395110 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:43.396840 (+ 1730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:43.396964 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:43.397105 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0925 22:38:43.397220 (+ 115us) server_negotiation.cc:299] Negotiation successful
0925 22:38:43.397328 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":166,"threads_started":1}
I20260925 22:38:43.400166 21308 tablet_service.cc:1467] Processing CreateTablet for tablet 4aa187919d7f4bbc873f94938cf4144c (DEFAULT_TABLE table=test-table [id=4f762b8b27fc4d9fa1063685dc1fd8b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:38:43.400166 21040 tablet_service.cc:1467] Processing CreateTablet for tablet 4aa187919d7f4bbc873f94938cf4144c (DEFAULT_TABLE table=test-table [id=4f762b8b27fc4d9fa1063685dc1fd8b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:38:43.400166 21174 tablet_service.cc:1467] Processing CreateTablet for tablet 4aa187919d7f4bbc873f94938cf4144c (DEFAULT_TABLE table=test-table [id=4f762b8b27fc4d9fa1063685dc1fd8b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:38:43.401470 21308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4aa187919d7f4bbc873f94938cf4144c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:38:43.401468 21040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4aa187919d7f4bbc873f94938cf4144c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:38:43.401468 21174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4aa187919d7f4bbc873f94938cf4144c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:38:43.410282 21392 tablet_bootstrap.cc:492] T 4aa187919d7f4bbc873f94938cf4144c P 6487020f6fdd432eaecee9f7c1c8beb1: Bootstrap starting.
I20260925 22:38:43.410282 21393 tablet_bootstrap.cc:492] T 4aa187919d7f4bbc873f94938cf4144c P 593624e63fab46768718c022af35acde: Bootstrap starting.
I20260925 22:38:43.411067 21391 tablet_bootstrap.cc:492] T 4aa187919d7f4bbc873f94938cf4144c P 2f130a10f1e64d5e8439fa19988dfee5: Bootstrap starting.
I20260925 22:38:43.413548 21393 tablet_bootstrap.cc:654] T 4aa187919d7f4bbc873f94938cf4144c P 593624e63fab46768718c022af35acde: Neither blocks nor log segments found. Creating new log.
I20260925 22:38:43.414073 21391 tablet_bootstrap.cc:654] T 4aa187919d7f4bbc873f94938cf4144c P 2f130a10f1e64d5e8439fa19988dfee5: Neither blocks nor log segments found. Creating new log.
I20260925 22:38:43.414153 21392 tablet_bootstrap.cc:654] T 4aa187919d7f4bbc873f94938cf4144c P 6487020f6fdd432eaecee9f7c1c8beb1: Neither blocks nor log segments found. Creating new log.
I20260925 22:38:43.414709 21393 log.cc:824] T 4aa187919d7f4bbc873f94938cf4144c P 593624e63fab46768718c022af35acde: Log is configured to *not* fsync() on all Append() calls
I20260925 22:38:43.414906 21391 log.cc:824] T 4aa187919d7f4bbc873f94938cf4144c P 2f130a10f1e64d5e8439fa19988dfee5: Log is configured to *not* fsync() on all Append() calls
I20260925 22:38:43.414911 21392 log.cc:824] T 4aa187919d7f4bbc873f94938cf4144c P 6487020f6fdd432eaecee9f7c1c8beb1: Log is configured to *not* fsync() on all Append() calls
I20260925 22:38:43.417035 21391 tablet_bootstrap.cc:492] T 4aa187919d7f4bbc873f94938cf4144c P 2f130a10f1e64d5e8439fa19988dfee5: No bootstrap required, opened a new log
I20260925 22:38:43.417040 21393 tablet_bootstrap.cc:492] T 4aa187919d7f4bbc873f94938cf4144c P 593624e63fab46768718c022af35acde: No bootstrap required, opened a new log
I20260925 22:38:43.417038 21392 tablet_bootstrap.cc:492] T 4aa187919d7f4bbc873f94938cf4144c P 6487020f6fdd432eaecee9f7c1c8beb1: No bootstrap required, opened a new log
I20260925 22:38:43.417253 21391 ts_tablet_manager.cc:1397] T 4aa187919d7f4bbc873f94938cf4144c P 2f130a10f1e64d5e8439fa19988dfee5: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 22:38:43.417248 21393 ts_tablet_manager.cc:1397] T 4aa187919d7f4bbc873f94938cf4144c P 593624e63fab46768718c022af35acde: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260925 22:38:43.417258 21392 ts_tablet_manager.cc:1397] T 4aa187919d7f4bbc873f94938cf4144c P 6487020f6fdd432eaecee9f7c1c8beb1: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260925 22:38:43.420832 21391 raft_consensus.cc:348] T 4aa187919d7f4bbc873f94938cf4144c P 2f130a10f1e64d5e8439fa19988dfee5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f130a10f1e64d5e8439fa19988dfee5" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43353 } } peers { permanent_uuid: "593624e63fab46768718c022af35acde" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 33913 } } peers { permanent_uuid: "6487020f6fdd432eaecee9f7c1c8beb1" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 45893 } }
I20260925 22:38:43.421092 21391 raft_consensus.cc:374] T 4aa187919d7f4bbc873f94938cf4144c P 2f130a10f1e64d5e8439fa19988dfee5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:38:43.421167 21391 raft_consensus.cc:729] T 4aa187919d7f4bbc873f94938cf4144c P 2f130a10f1e64d5e8439fa19988dfee5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f130a10f1e64d5e8439fa19988dfee5, State: Initialized, Role: FOLLOWER
I20260925 22:38:43.421665 21391 consensus_queue.cc:260] T 4aa187919d7f4bbc873f94938cf4144c P 2f130a10f1e64d5e8439fa19988dfee5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f130a10f1e64d5e8439fa19988dfee5" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43353 } } peers { permanent_uuid: "593624e63fab46768718c022af35acde" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 33913 } } peers { permanent_uuid: "6487020f6fdd432eaecee9f7c1c8beb1" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 45893 } }
I20260925 22:38:43.421677 21392 raft_consensus.cc:348] T 4aa187919d7f4bbc873f94938cf4144c P 6487020f6fdd432eaecee9f7c1c8beb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f130a10f1e64d5e8439fa19988dfee5" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43353 } } peers { permanent_uuid: "593624e63fab46768718c022af35acde" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 33913 } } peers { permanent_uuid: "6487020f6fdd432eaecee9f7c1c8beb1" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 45893 } }
I20260925 22:38:43.422047 21392 raft_consensus.cc:374] T 4aa187919d7f4bbc873f94938cf4144c P 6487020f6fdd432eaecee9f7c1c8beb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:38:43.422130 21392 raft_consensus.cc:729] T 4aa187919d7f4bbc873f94938cf4144c P 6487020f6fdd432eaecee9f7c1c8beb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6487020f6fdd432eaecee9f7c1c8beb1, State: Initialized, Role: FOLLOWER
I20260925 22:38:43.422451 21374 heartbeater.cc:503] Master 127.6.246.254:43017 was elected leader, sending a full tablet report...
I20260925 22:38:43.422242 21393 raft_consensus.cc:348] T 4aa187919d7f4bbc873f94938cf4144c P 593624e63fab46768718c022af35acde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f130a10f1e64d5e8439fa19988dfee5" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43353 } } peers { permanent_uuid: "593624e63fab46768718c022af35acde" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 33913 } } peers { permanent_uuid: "6487020f6fdd432eaecee9f7c1c8beb1" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 45893 } }
I20260925 22:38:43.422545 21393 raft_consensus.cc:374] T 4aa187919d7f4bbc873f94938cf4144c P 593624e63fab46768718c022af35acde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:38:43.422616 21393 raft_consensus.cc:729] T 4aa187919d7f4bbc873f94938cf4144c P 593624e63fab46768718c022af35acde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 593624e63fab46768718c022af35acde, State: Initialized, Role: FOLLOWER
I20260925 22:38:43.422684 21392 consensus_queue.cc:260] T 4aa187919d7f4bbc873f94938cf4144c P 6487020f6fdd432eaecee9f7c1c8beb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f130a10f1e64d5e8439fa19988dfee5" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43353 } } peers { permanent_uuid: "593624e63fab46768718c022af35acde" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 33913 } } peers { permanent_uuid: "6487020f6fdd432eaecee9f7c1c8beb1" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 45893 } }
I20260925 22:38:43.422832 21391 ts_tablet_manager.cc:1428] T 4aa187919d7f4bbc873f94938cf4144c P 2f130a10f1e64d5e8439fa19988dfee5: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260925 22:38:43.423126 21393 consensus_queue.cc:260] T 4aa187919d7f4bbc873f94938cf4144c P 593624e63fab46768718c022af35acde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f130a10f1e64d5e8439fa19988dfee5" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43353 } } peers { permanent_uuid: "593624e63fab46768718c022af35acde" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 33913 } } peers { permanent_uuid: "6487020f6fdd432eaecee9f7c1c8beb1" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 45893 } }
I20260925 22:38:43.423593 21240 heartbeater.cc:503] Master 127.6.246.254:43017 was elected leader, sending a full tablet report...
I20260925 22:38:43.424145 21392 ts_tablet_manager.cc:1428] T 4aa187919d7f4bbc873f94938cf4144c P 6487020f6fdd432eaecee9f7c1c8beb1: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260925 22:38:43.424525 21393 ts_tablet_manager.cc:1428] T 4aa187919d7f4bbc873f94938cf4144c P 593624e63fab46768718c022af35acde: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260925 22:38:43.425042 21106 heartbeater.cc:503] Master 127.6.246.254:43017 was elected leader, sending a full tablet report...
W20260925 22:38:43.555253 21107 tablet.cc:2369] T 4aa187919d7f4bbc873f94938cf4144c P 593624e63fab46768718c022af35acde: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 22:38:43.571892 21241 tablet.cc:2369] T 4aa187919d7f4bbc873f94938cf4144c P 6487020f6fdd432eaecee9f7c1c8beb1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 22:38:43.585101 21375 tablet.cc:2369] T 4aa187919d7f4bbc873f94938cf4144c P 2f130a10f1e64d5e8439fa19988dfee5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:38:43.723718 21398 raft_consensus.cc:482] T 4aa187919d7f4bbc873f94938cf4144c P 6487020f6fdd432eaecee9f7c1c8beb1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:38:43.723965 21398 raft_consensus.cc:504] T 4aa187919d7f4bbc873f94938cf4144c P 6487020f6fdd432eaecee9f7c1c8beb1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f130a10f1e64d5e8439fa19988dfee5" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43353 } } peers { permanent_uuid: "593624e63fab46768718c022af35acde" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 33913 } } peers { permanent_uuid: "6487020f6fdd432eaecee9f7c1c8beb1" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 45893 } }
I20260925 22:38:43.725273 21398 leader_election.cc:288] T 4aa187919d7f4bbc873f94938cf4144c P 6487020f6fdd432eaecee9f7c1c8beb1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2f130a10f1e64d5e8439fa19988dfee5 (127.6.246.195:43353), 593624e63fab46768718c022af35acde (127.6.246.193:33913)
I20260925 22:38:43.729964 21386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:43.725715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.194:48845 (local address 127.6.246.195:43353)
0925 22:38:43.725895 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 22:38:43.725903 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 22:38:43.726005 (+ 102us) server_negotiation.cc:407] Connection header received
0925 22:38:43.726213 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:38:43.726218 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:38:43.726273 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:38:43.726362 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:38:43.726835 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:43.727689 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:43.728932 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:43.729389 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:43.729518 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:43.729577 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 22:38:43.729758 (+ 181us) server_negotiation.cc:299] Negotiation successful
0925 22:38:43.729833 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 22:38:43.729979 21400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:43.725609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.195:43353 (local address 127.6.246.194:48845)
0925 22:38:43.725891 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:43.725907 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:38:43.726069 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:38:43.726532 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:38:43.726537 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:38:43.726558 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:38:43.726709 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:38:43.726715 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:38:43.727847 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:38:43.727850 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:38:43.728802 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:38:43.728812 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:38:43.729511 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:38:43.729515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:38:43.729614 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:43.729639 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 22:38:43.729732 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 22:38:43.729811 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":60,"thread_start_us":86,"threads_started":1}
I20260925 22:38:43.730098 21401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:43.725990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.193:33913 (local address 127.6.246.194:47877)
0925 22:38:43.726239 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:43.726250 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 22:38:43.726340 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:38:43.726608 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:38:43.726611 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:38:43.726627 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:38:43.726758 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:38:43.726762 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:38:43.727837 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:38:43.727840 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:38:43.728813 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:38:43.728822 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:38:43.729607 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:38:43.729611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:38:43.729711 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:43.729736 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 22:38:43.729840 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 22:38:43.729912 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":25,"thread_start_us":81,"threads_started":1}
I20260925 22:38:43.730083 21390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:43.726160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.194:47877 (local address 127.6.246.193:33913)
0925 22:38:43.726303 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 22:38:43.726310 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:38:43.726324 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:38:43.726424 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:38:43.726427 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:38:43.726482 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:38:43.726560 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:38:43.726913 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:43.727689 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:43.728933 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:43.729495 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:43.729619 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:43.729683 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 22:38:43.729868 (+ 185us) server_negotiation.cc:299] Negotiation successful
0925 22:38:43.729946 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 22:38:43.730647 21060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4aa187919d7f4bbc873f94938cf4144c" candidate_uuid: "6487020f6fdd432eaecee9f7c1c8beb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "593624e63fab46768718c022af35acde" is_pre_election: true
I20260925 22:38:43.730647 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4aa187919d7f4bbc873f94938cf4144c" candidate_uuid: "6487020f6fdd432eaecee9f7c1c8beb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f130a10f1e64d5e8439fa19988dfee5" is_pre_election: true
I20260925 22:38:43.730979 21328 raft_consensus.cc:2454] T 4aa187919d7f4bbc873f94938cf4144c P 2f130a10f1e64d5e8439fa19988dfee5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6487020f6fdd432eaecee9f7c1c8beb1 in term 0.
I20260925 22:38:43.730979 21060 raft_consensus.cc:2454] T 4aa187919d7f4bbc873f94938cf4144c P 593624e63fab46768718c022af35acde [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6487020f6fdd432eaecee9f7c1c8beb1 in term 0.
I20260925 22:38:43.731468 21128 leader_election.cc:302] T 4aa187919d7f4bbc873f94938cf4144c P 6487020f6fdd432eaecee9f7c1c8beb1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f130a10f1e64d5e8439fa19988dfee5, 6487020f6fdd432eaecee9f7c1c8beb1; no voters:
I20260925 22:38:43.731730 21398 raft_consensus.cc:2781] T 4aa187919d7f4bbc873f94938cf4144c P 6487020f6fdd432eaecee9f7c1c8beb1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:38:43.731824 21398 raft_consensus.cc:482] T 4aa187919d7f4bbc873f94938cf4144c P 6487020f6fdd432eaecee9f7c1c8beb1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:38:43.731868 21398 raft_consensus.cc:3037] T 4aa187919d7f4bbc873f94938cf4144c P 6487020f6fdd432eaecee9f7c1c8beb1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:38:43.733047 21398 raft_consensus.cc:504] T 4aa187919d7f4bbc873f94938cf4144c P 6487020f6fdd432eaecee9f7c1c8beb1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f130a10f1e64d5e8439fa19988dfee5" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43353 } } peers { permanent_uuid: "593624e63fab46768718c022af35acde" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 33913 } } peers { permanent_uuid: "6487020f6fdd432eaecee9f7c1c8beb1" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 45893 } }
I20260925 22:38:43.733387 21398 leader_election.cc:288] T 4aa187919d7f4bbc873f94938cf4144c P 6487020f6fdd432eaecee9f7c1c8beb1 [CANDIDATE]: Term 1 election: Requested vote from peers 2f130a10f1e64d5e8439fa19988dfee5 (127.6.246.195:43353), 593624e63fab46768718c022af35acde (127.6.246.193:33913)
I20260925 22:38:43.733749 21060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4aa187919d7f4bbc873f94938cf4144c" candidate_uuid: "6487020f6fdd432eaecee9f7c1c8beb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "593624e63fab46768718c022af35acde"
I20260925 22:38:43.733749 21328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4aa187919d7f4bbc873f94938cf4144c" candidate_uuid: "6487020f6fdd432eaecee9f7c1c8beb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f130a10f1e64d5e8439fa19988dfee5"
I20260925 22:38:43.733919 21060 raft_consensus.cc:3037] T 4aa187919d7f4bbc873f94938cf4144c P 593624e63fab46768718c022af35acde [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:38:43.733920 21328 raft_consensus.cc:3037] T 4aa187919d7f4bbc873f94938cf4144c P 2f130a10f1e64d5e8439fa19988dfee5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:38:43.735287 21328 raft_consensus.cc:2454] T 4aa187919d7f4bbc873f94938cf4144c P 2f130a10f1e64d5e8439fa19988dfee5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6487020f6fdd432eaecee9f7c1c8beb1 in term 1.
I20260925 22:38:43.735287 21060 raft_consensus.cc:2454] T 4aa187919d7f4bbc873f94938cf4144c P 593624e63fab46768718c022af35acde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6487020f6fdd432eaecee9f7c1c8beb1 in term 1.
I20260925 22:38:43.735576 21128 leader_election.cc:302] T 4aa187919d7f4bbc873f94938cf4144c P 6487020f6fdd432eaecee9f7c1c8beb1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f130a10f1e64d5e8439fa19988dfee5, 6487020f6fdd432eaecee9f7c1c8beb1; no voters:
I20260925 22:38:43.735731 21398 raft_consensus.cc:2781] T 4aa187919d7f4bbc873f94938cf4144c P 6487020f6fdd432eaecee9f7c1c8beb1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:38:43.735981 21398 raft_consensus.cc:686] T 4aa187919d7f4bbc873f94938cf4144c P 6487020f6fdd432eaecee9f7c1c8beb1 [term 1 LEADER]: Becoming Leader. State: Replica: 6487020f6fdd432eaecee9f7c1c8beb1, State: Running, Role: LEADER
I20260925 22:38:43.736339 21398 consensus_queue.cc:237] T 4aa187919d7f4bbc873f94938cf4144c P 6487020f6fdd432eaecee9f7c1c8beb1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f130a10f1e64d5e8439fa19988dfee5" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43353 } } peers { permanent_uuid: "593624e63fab46768718c022af35acde" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 33913 } } peers { permanent_uuid: "6487020f6fdd432eaecee9f7c1c8beb1" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 45893 } }
I20260925 22:38:43.739792 20913 catalog_manager.cc:5495] T 4aa187919d7f4bbc873f94938cf4144c P 6487020f6fdd432eaecee9f7c1c8beb1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6487020f6fdd432eaecee9f7c1c8beb1 (127.6.246.194). New cstate: current_term: 1 leader_uuid: "6487020f6fdd432eaecee9f7c1c8beb1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f130a10f1e64d5e8439fa19988dfee5" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "593624e63fab46768718c022af35acde" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 33913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6487020f6fdd432eaecee9f7c1c8beb1" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 45893 } health_report { overall_health: HEALTHY } } }
I20260925 22:38:43.759150 21389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:43.755154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59166 (local address 127.6.246.194:45893)
0925 22:38:43.755313 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 22:38:43.755318 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:38:43.755334 (+ 16us) server_negotiation.cc:407] Connection header received
0925 22:38:43.755509 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:38:43.755512 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:38:43.755573 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:38:43.755694 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:38:43.755969 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:43.756595 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:43.757535 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:43.757781 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:43.757907 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:43.758234 (+ 327us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:38:43.758718 (+ 484us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:38:43.758874 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 22:38:43.758920 (+ 46us) server_negotiation.cc:299] Negotiation successful
0925 22:38:43.758979 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 22:38:43.765919 21060 raft_consensus.cc:1261] T 4aa187919d7f4bbc873f94938cf4144c P 593624e63fab46768718c022af35acde [term 1 FOLLOWER]: Refusing update from remote peer 6487020f6fdd432eaecee9f7c1c8beb1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:38:43.765944 21328 raft_consensus.cc:1261] T 4aa187919d7f4bbc873f94938cf4144c P 2f130a10f1e64d5e8439fa19988dfee5 [term 1 FOLLOWER]: Refusing update from remote peer 6487020f6fdd432eaecee9f7c1c8beb1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:38:43.766732 21402 consensus_queue.cc:1035] T 4aa187919d7f4bbc873f94938cf4144c P 6487020f6fdd432eaecee9f7c1c8beb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "593624e63fab46768718c022af35acde" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:38:43.766932 21398 consensus_queue.cc:1035] T 4aa187919d7f4bbc873f94938cf4144c P 6487020f6fdd432eaecee9f7c1c8beb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f130a10f1e64d5e8439fa19988dfee5" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:38:43.773561 21405 mvcc.cc:205] Tried to move back new op lower bound from 7333379783736991744 to 7333379783626825728. Current Snapshot: MvccSnapshot[applied={T|T < 7333379783736991744}]
I20260925 22:38:43.777691 21407 mvcc.cc:205] Tried to move back new op lower bound from 7333379783736991744 to 7333379783626825728. Current Snapshot: MvccSnapshot[applied={T|T < 7333379783736991744}]
I20260925 22:38:43.784225 20914 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56144:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:38:43.784384 20914 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56144:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:38:43.786799 20914 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 761938f0951e48a0910db98fde88b05f: Sending DeleteTablet for 3 replicas of tablet 4aa187919d7f4bbc873f94938cf4144c
I20260925 22:38:43.787725 21174 tablet_service.cc:1514] Processing DeleteTablet for tablet 4aa187919d7f4bbc873f94938cf4144c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:38:43 UTC) from {username='kudu', principal='kudu/127.6.246.254@KRBTEST.COM'} at 127.0.0.1:59156
I20260925 22:38:43.787739 21308 tablet_service.cc:1514] Processing DeleteTablet for tablet 4aa187919d7f4bbc873f94938cf4144c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:38:43 UTC) from {username='kudu', principal='kudu/127.6.246.254@KRBTEST.COM'} at 127.0.0.1:55248
I20260925 22:38:43.787886 21040 tablet_service.cc:1514] Processing DeleteTablet for tablet 4aa187919d7f4bbc873f94938cf4144c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:38:43 UTC) from {username='kudu', principal='kudu/127.6.246.254@KRBTEST.COM'} at 127.0.0.1:49904
I20260925 22:38:43.787937 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 20977
I20260925 22:38:43.788898 21415 tablet_replica.cc:330] T 4aa187919d7f4bbc873f94938cf4144c P 6487020f6fdd432eaecee9f7c1c8beb1: stopping tablet replica
I20260925 22:38:43.789108 21416 tablet_replica.cc:330] T 4aa187919d7f4bbc873f94938cf4144c P 2f130a10f1e64d5e8439fa19988dfee5: stopping tablet replica
I20260925 22:38:43.789233 21415 raft_consensus.cc:2229] T 4aa187919d7f4bbc873f94938cf4144c P 6487020f6fdd432eaecee9f7c1c8beb1 [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:38:43.789358 21416 raft_consensus.cc:2229] T 4aa187919d7f4bbc873f94938cf4144c P 2f130a10f1e64d5e8439fa19988dfee5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:38:43.789600 21415 raft_consensus.cc:2258] T 4aa187919d7f4bbc873f94938cf4144c P 6487020f6fdd432eaecee9f7c1c8beb1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:38:43.789656 21416 raft_consensus.cc:2258] T 4aa187919d7f4bbc873f94938cf4144c P 2f130a10f1e64d5e8439fa19988dfee5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:38:43.791096 21415 ts_tablet_manager.cc:1905] T 4aa187919d7f4bbc873f94938cf4144c P 6487020f6fdd432eaecee9f7c1c8beb1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:38:43.791096 21416 ts_tablet_manager.cc:1905] T 4aa187919d7f4bbc873f94938cf4144c P 2f130a10f1e64d5e8439fa19988dfee5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:38:43.794014 21416 ts_tablet_manager.cc:1918] T 4aa187919d7f4bbc873f94938cf4144c P 2f130a10f1e64d5e8439fa19988dfee5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:38:43.794027 21415 ts_tablet_manager.cc:1918] T 4aa187919d7f4bbc873f94938cf4144c P 6487020f6fdd432eaecee9f7c1c8beb1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:38:43.794126 21415 log.cc:1196] T 4aa187919d7f4bbc873f94938cf4144c P 6487020f6fdd432eaecee9f7c1c8beb1: Deleting WAL directory at /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-1/wal/wals/4aa187919d7f4bbc873f94938cf4144c
I20260925 22:38:43.794224 21416 log.cc:1196] T 4aa187919d7f4bbc873f94938cf4144c P 2f130a10f1e64d5e8439fa19988dfee5: Deleting WAL directory at /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-2/wal/wals/4aa187919d7f4bbc873f94938cf4144c
I20260925 22:38:43.794500 21415 ts_tablet_manager.cc:1939] T 4aa187919d7f4bbc873f94938cf4144c P 6487020f6fdd432eaecee9f7c1c8beb1: Deleting consensus metadata
I20260925 22:38:43.794657 21416 ts_tablet_manager.cc:1939] T 4aa187919d7f4bbc873f94938cf4144c P 2f130a10f1e64d5e8439fa19988dfee5: Deleting consensus metadata
I20260925 22:38:43.795831 20900 catalog_manager.cc:4841] TS 2f130a10f1e64d5e8439fa19988dfee5 (127.6.246.195:43353): tablet 4aa187919d7f4bbc873f94938cf4144c (table test-table [id=4f762b8b27fc4d9fa1063685dc1fd8b2]) successfully deleted
W20260925 22:38:43.796200 20900 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.246.193:33913 (error 108)
I20260925 22:38:43.796511 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 21111
I20260925 22:38:43.795831 20901 catalog_manager.cc:4841] TS 6487020f6fdd432eaecee9f7c1c8beb1 (127.6.246.194:45893): tablet 4aa187919d7f4bbc873f94938cf4144c (table test-table [id=4f762b8b27fc4d9fa1063685dc1fd8b2]) successfully deleted
I20260925 22:38:43.798120 21388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:43.797846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.193:33913 (local address 127.0.0.1:49916)
0925 22:38:43.797974 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:43.798045 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.193:33913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 22:38:43.798316 20900 catalog_manager.cc:4584] TS 593624e63fab46768718c022af35acde (127.6.246.193:33913): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4aa187919d7f4bbc873f94938cf4144c: Network error: Client connection negotiation failed: client connection to 127.6.246.193:33913: connect: Connection refused (error 111)
I20260925 22:38:43.803387 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 21245
I20260925 22:38:43.809461 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 20883
I20260925 22:38:43.816218 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:43017
--webserver_interface=127.6.246.254
--webserver_port=33505
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab
--principal=kudu/127.6.246.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:34883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.246.254:43017
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:38:43.935773 21418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:38:43.936069 21418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:38:43.936137 21418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:38:43.940202 21418 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:38:43.940289 21418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:38:43.940327 21418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:38:43.940384 21418 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:38:43.940423 21418 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:38:43.945814 21418 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:34883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.6.246.254:43017
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab
--principal=kudu/127.6.246.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:43017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.246.254
--webserver_port=33505
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21418
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:38:43.946885 21418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:38:43.947932 21418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:38:43.961179 21425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:38:43.961266 21418 server_base.cc:1034] running on GCE node
W20260925 22:38:43.961274 21427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:38:43.961161 21424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:38:43.962033 21418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:38:43.963418 21418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:38:43.964534 21418 hybrid_clock.cc:648] HybridClock initialized: now 1790375923964521 us; error 34 us; skew 500 ppm
Sep 25 22:38:43 dist-test-slave-wjdr krb5kdc[20870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375923, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:38:43.967482 21418 init.cc:377] Logged in from keytab as kudu/127.6.246.254@KRBTEST.COM (short username kudu)
I20260925 22:38:43.968425 21418 webserver.cc:460] Webserver started at http://127.6.246.254:33505/ using document root <none> and password file <none>
I20260925 22:38:43.969087 21418 fs_manager.cc:359] Metadata directory not provided
I20260925 22:38:43.969192 21418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:38:43.973249 21418 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:38:43.975569 21434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:38:43.977078 21418 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:38:43.977221 21418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/master-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/master-0/wal
uuid: "761938f0951e48a0910db98fde88b05f"
format_stamp: "Formatted at 2026-09-25 22:38:42 on dist-test-slave-wjdr"
server_key: "18768535b2555fb849ed97cf45aebc3c"
server_key_iv: "5804c09b287daae23dfb0a1be6d786f4"
server_key_version: "encryptionkey@0"
I20260925 22:38:43.977839 21418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:38:43.999580 21418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:38:44.003132 21418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:38:44.003357 21418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:38:44.010655 21418 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.254:43017
I20260925 22:38:44.010977 21486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.254:43017 every 8 connection(s)
I20260925 22:38:44.011790 21418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
I20260925 22:38:44.013208 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 21418
I20260925 22:38:44.013680 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.193:33913
--local_ip_for_outbound_sockets=127.6.246.193
--tserver_master_addrs=127.6.246.254:43017
--webserver_port=38055
--webserver_interface=127.6.246.193
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.193.keytab
--principal=kudu/127.6.246.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:34883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:38:44.017376 21487 sys_catalog.cc:263] Verifying existing consensus state
I20260925 22:38:44.020143 21487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 761938f0951e48a0910db98fde88b05f: Bootstrap starting.
I20260925 22:38:44.042171 21487 log.cc:824] T 00000000000000000000000000000000 P 761938f0951e48a0910db98fde88b05f: Log is configured to *not* fsync() on all Append() calls
I20260925 22:38:44.053479 21487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 761938f0951e48a0910db98fde88b05f: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 22:38:44.053910 21487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 761938f0951e48a0910db98fde88b05f: Bootstrap complete.
I20260925 22:38:44.058084 21487 raft_consensus.cc:348] T 00000000000000000000000000000000 P 761938f0951e48a0910db98fde88b05f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "761938f0951e48a0910db98fde88b05f" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 43017 } }
I20260925 22:38:44.058467 21487 raft_consensus.cc:729] T 00000000000000000000000000000000 P 761938f0951e48a0910db98fde88b05f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 761938f0951e48a0910db98fde88b05f, State: Initialized, Role: FOLLOWER
I20260925 22:38:44.058964 21487 consensus_queue.cc:260] T 00000000000000000000000000000000 P 761938f0951e48a0910db98fde88b05f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "761938f0951e48a0910db98fde88b05f" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 43017 } }
I20260925 22:38:44.059078 21487 raft_consensus.cc:388] T 00000000000000000000000000000000 P 761938f0951e48a0910db98fde88b05f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:38:44.059168 21487 raft_consensus.cc:482] T 00000000000000000000000000000000 P 761938f0951e48a0910db98fde88b05f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:38:44.059283 21487 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 761938f0951e48a0910db98fde88b05f [term 1 FOLLOWER]: Advancing to term 2
I20260925 22:38:44.061265 21487 raft_consensus.cc:504] T 00000000000000000000000000000000 P 761938f0951e48a0910db98fde88b05f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "761938f0951e48a0910db98fde88b05f" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 43017 } }
I20260925 22:38:44.061628 21487 leader_election.cc:302] T 00000000000000000000000000000000 P 761938f0951e48a0910db98fde88b05f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 761938f0951e48a0910db98fde88b05f; no voters:
I20260925 22:38:44.061961 21487 leader_election.cc:288] T 00000000000000000000000000000000 P 761938f0951e48a0910db98fde88b05f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 22:38:44.062057 21491 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 761938f0951e48a0910db98fde88b05f [term 2 FOLLOWER]: Leader election won for term 2
I20260925 22:38:44.062309 21491 raft_consensus.cc:686] T 00000000000000000000000000000000 P 761938f0951e48a0910db98fde88b05f [term 2 LEADER]: Becoming Leader. State: Replica: 761938f0951e48a0910db98fde88b05f, State: Running, Role: LEADER
I20260925 22:38:44.062613 21491 consensus_queue.cc:237] T 00000000000000000000000000000000 P 761938f0951e48a0910db98fde88b05f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "761938f0951e48a0910db98fde88b05f" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 43017 } }
I20260925 22:38:44.063290 21487 sys_catalog.cc:564] T 00000000000000000000000000000000 P 761938f0951e48a0910db98fde88b05f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:38:44.063583 21491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 761938f0951e48a0910db98fde88b05f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 761938f0951e48a0910db98fde88b05f. Latest consensus state: current_term: 2 leader_uuid: "761938f0951e48a0910db98fde88b05f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "761938f0951e48a0910db98fde88b05f" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 43017 } } }
I20260925 22:38:44.063704 21491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 761938f0951e48a0910db98fde88b05f [sys.catalog]: This master's current role is: LEADER
I20260925 22:38:44.064062 21495 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:38:44.064244 21492 sys_catalog.cc:455] T 00000000000000000000000000000000 P 761938f0951e48a0910db98fde88b05f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "761938f0951e48a0910db98fde88b05f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "761938f0951e48a0910db98fde88b05f" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 43017 } } }
I20260925 22:38:44.064431 21492 sys_catalog.cc:458] T 00000000000000000000000000000000 P 761938f0951e48a0910db98fde88b05f [sys.catalog]: This master's current role is: LEADER
I20260925 22:38:44.069111 21495 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:38:44.069857 21495 catalog_manager.cc:1259] Loaded cluster ID: 79b9eef78b7641a9a88d8832afd1e153
I20260925 22:38:44.069932 21495 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:38:44.071044 21495 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260925 22:38:44.073899 21495 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260925 22:38:44.078598 21495 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:38:44.079331 21495 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 761938f0951e48a0910db98fde88b05f: Loaded TSK: 0
I20260925 22:38:44.080308 21495 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 22:38:44.153331 21489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:38:44.153609 21489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:38:44.153681 21489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:38:44.157987 21489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:38:44.158064 21489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:38:44.158145 21489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.193
I20260925 22:38:44.163630 21489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:34883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.193.keytab
--principal=kudu/127.6.246.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.193:33913
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.246.193
--webserver_port=38055
--tserver_master_addrs=127.6.246.254:43017
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.193
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:38:44.164961 21489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:38:44.166030 21489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:38:44.175473 21515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:38:44.175422 21517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:38:44.175444 21514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:38:44.175877 21489 server_base.cc:1034] running on GCE node
I20260925 22:38:44.176291 21489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:38:44.176967 21489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:38:44.178169 21489 hybrid_clock.cc:648] HybridClock initialized: now 1790375924178144 us; error 42 us; skew 500 ppm
Sep 25 22:38:44 dist-test-slave-wjdr krb5kdc[20870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375924, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:38:44.181169 21489 init.cc:377] Logged in from keytab as kudu/127.6.246.193@KRBTEST.COM (short username kudu)
I20260925 22:38:44.182228 21489 webserver.cc:460] Webserver started at http://127.6.246.193:38055/ using document root <none> and password file <none>
I20260925 22:38:44.182803 21489 fs_manager.cc:359] Metadata directory not provided
I20260925 22:38:44.182906 21489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:38:44.186867 21489 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:38:44.189046 21524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:38:44.190488 21489 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:38:44.190626 21489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-0/wal
uuid: "593624e63fab46768718c022af35acde"
format_stamp: "Formatted at 2026-09-25 22:38:42 on dist-test-slave-wjdr"
server_key: "65166019af2a3076ec1047c39d91369d"
server_key_iv: "abccb242a4c3c976be25e55b993397a4"
server_key_version: "encryptionkey@0"
I20260925 22:38:44.191522 21489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:38:44.220886 21489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:38:44.224202 21489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:38:44.224416 21489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:38:44.225082 21489 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:38:44.226127 21531 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 22:38:44.230347 21489 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 22:38:44.230465 21489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260925 22:38:44.230557 21489 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 22:38:44.232395 21489 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 22:38:44.232466 21489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260925 22:38:44.232756 21531 tablet_bootstrap.cc:492] T 4aa187919d7f4bbc873f94938cf4144c P 593624e63fab46768718c022af35acde: Bootstrap starting.
I20260925 22:38:44.244028 21489 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.193:33913
I20260925 22:38:44.244124 21638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.193:33913 every 8 connection(s)
I20260925 22:38:44.245139 21489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
I20260925 22:38:44.251653 21531 log.cc:824] T 4aa187919d7f4bbc873f94938cf4144c P 593624e63fab46768718c022af35acde: Log is configured to *not* fsync() on all Append() calls
I20260925 22:38:44.251881 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 21489
I20260925 22:38:44.252485 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.194:45893
--local_ip_for_outbound_sockets=127.6.246.194
--tserver_master_addrs=127.6.246.254:43017
--webserver_port=36019
--webserver_interface=127.6.246.194
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.194.keytab
--principal=kudu/127.6.246.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:34883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 22:38:44 dist-test-slave-wjdr krb5kdc[20870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375924, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.193@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
I20260925 22:38:44.260639 21531 tablet_bootstrap.cc:492] T 4aa187919d7f4bbc873f94938cf4144c P 593624e63fab46768718c022af35acde: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 22:38:44.261128 21531 tablet_bootstrap.cc:492] T 4aa187919d7f4bbc873f94938cf4144c P 593624e63fab46768718c022af35acde: Bootstrap complete.
I20260925 22:38:44.262674 21531 ts_tablet_manager.cc:1397] T 4aa187919d7f4bbc873f94938cf4144c P 593624e63fab46768718c022af35acde: Time spent bootstrapping tablet: real 0.030s user 0.027s sys 0.000s
I20260925 22:38:44.264088 21641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:44.247340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.193:36555 (local address 127.6.246.254:43017)
0925 22:38:44.247769 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0925 22:38:44.247793 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0925 22:38:44.248085 (+ 292us) server_negotiation.cc:407] Connection header received
0925 22:38:44.249405 (+ 1320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:38:44.249424 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:38:44.249846 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:38:44.250224 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:38:44.251300 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:44.252638 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:44.254518 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:44.255184 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:44.255361 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:44.258383 (+ 3022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:38:44.258388 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:38:44.258401 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:38:44.258430 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:38:44.260506 (+ 2076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:38:44.261127 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:38:44.261133 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:38:44.261139 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:38:44.261320 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:38:44.261688 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:38:44.261693 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:38:44.261695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:38:44.262067 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:38:44.262216 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 22:38:44.262775 (+ 559us) server_negotiation.cc:299] Negotiation successful
0925 22:38:44.263070 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":124,"threads_started":1}
I20260925 22:38:44.264155 21642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:44.247225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43017 (local address 127.6.246.193:36555)
0925 22:38:44.247896 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:44.247952 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 22:38:44.248893 (+ 941us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:38:44.250268 (+ 1375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:38:44.250275 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:38:44.250648 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:38:44.251113 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:38:44.251127 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:38:44.252840 (+ 1713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:38:44.252848 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:38:44.254372 (+ 1524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:38:44.254383 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:38:44.255579 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:38:44.255586 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:38:44.255840 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:44.256551 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:38:44.256568 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:38:44.258155 (+ 1587us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:38:44.260635 (+ 2480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:38:44.260641 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:38:44.260654 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:38:44.261001 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:38:44.261429 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:38:44.261434 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:38:44.261436 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:38:44.261586 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:38:44.262205 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:38:44.262211 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:38:44.262455 (+ 244us) client_negotiation.cc:769] Sending connection context
0925 22:38:44.262656 (+ 201us) client_negotiation.cc:240] Negotiation successful
0925 22:38:44.263099 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":153,"threads_started":1}
I20260925 22:38:44.266866 21639 heartbeater.cc:347] Connected to a master server at 127.6.246.254:43017
I20260925 22:38:44.267124 21639 heartbeater.cc:464] Registering TS with master...
I20260925 22:38:44.267009 21531 raft_consensus.cc:348] T 4aa187919d7f4bbc873f94938cf4144c P 593624e63fab46768718c022af35acde [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f130a10f1e64d5e8439fa19988dfee5" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43353 } } peers { permanent_uuid: "593624e63fab46768718c022af35acde" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 33913 } } peers { permanent_uuid: "6487020f6fdd432eaecee9f7c1c8beb1" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 45893 } }
I20260925 22:38:44.267535 21531 raft_consensus.cc:729] T 4aa187919d7f4bbc873f94938cf4144c P 593624e63fab46768718c022af35acde [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 593624e63fab46768718c022af35acde, State: Initialized, Role: FOLLOWER
I20260925 22:38:44.267690 21639 heartbeater.cc:511] Master 127.6.246.254:43017 requested a full tablet report, sending...
I20260925 22:38:44.268033 21531 consensus_queue.cc:260] T 4aa187919d7f4bbc873f94938cf4144c P 593624e63fab46768718c022af35acde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f130a10f1e64d5e8439fa19988dfee5" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43353 } } peers { permanent_uuid: "593624e63fab46768718c022af35acde" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 33913 } } peers { permanent_uuid: "6487020f6fdd432eaecee9f7c1c8beb1" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 45893 } }
I20260925 22:38:44.269090 21531 ts_tablet_manager.cc:1428] T 4aa187919d7f4bbc873f94938cf4144c P 593624e63fab46768718c022af35acde: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260925 22:38:44.270597 21451 ts_manager.cc:194] Registered new tserver with Master: 593624e63fab46768718c022af35acde (127.6.246.193:33913)
I20260925 22:38:44.274780 21451 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.246.193@KRBTEST.COM'} at 127.6.246.193:36555
I20260925 22:38:44.277639 21639 heartbeater.cc:503] Master 127.6.246.254:43017 was elected leader, sending a full tablet report...
Sep 25 22:38:44 dist-test-slave-wjdr krb5kdc[20870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375923, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.254@KRBTEST.COM for kudu/127.6.246.193@KRBTEST.COM
I20260925 22:38:44.285274 21646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:44.273603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.193:33913 (local address 127.0.0.1:49924)
0925 22:38:44.274136 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:44.274189 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 22:38:44.274322 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:38:44.275054 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:38:44.275060 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:38:44.275514 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:38:44.275649 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:38:44.275657 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:38:44.277164 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:38:44.277167 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:38:44.278175 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:38:44.278187 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:38:44.278949 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:38:44.278953 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:38:44.279070 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:44.279691 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:38:44.279704 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:38:44.281271 (+ 1567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:38:44.283539 (+ 2268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:38:44.283547 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:38:44.283552 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:38:44.283790 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:38:44.284127 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:38:44.284129 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:38:44.284131 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:38:44.284210 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:38:44.284731 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:38:44.284736 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:38:44.284847 (+ 111us) client_negotiation.cc:769] Sending connection context
0925 22:38:44.284975 (+ 128us) client_negotiation.cc:240] Negotiation successful
0925 22:38:44.285132 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":205,"threads_started":1}
I20260925 22:38:44.285429 21647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:44.273675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49924 (local address 127.6.246.193:33913)
0925 22:38:44.274497 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0925 22:38:44.274508 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 22:38:44.274532 (+ 24us) server_negotiation.cc:407] Connection header received
0925 22:38:44.274598 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:38:44.274604 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:38:44.274810 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:38:44.274984 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:38:44.275775 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:44.277023 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:44.278321 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:44.278805 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:44.278924 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:44.281453 (+ 2529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:38:44.281462 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:38:44.281475 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:38:44.281502 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:38:44.283430 (+ 1928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:38:44.283915 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:38:44.283920 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:38:44.283922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:38:44.284033 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:38:44.284295 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:38:44.284298 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:38:44.284299 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:38:44.284610 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:38:44.284735 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 22:38:44.285076 (+ 341us) server_negotiation.cc:299] Negotiation successful
0925 22:38:44.285243 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":723,"thread_start_us":169,"threads_started":1}
I20260925 22:38:44.286396 21573 tablet_service.cc:1514] Processing DeleteTablet for tablet 4aa187919d7f4bbc873f94938cf4144c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:38:43 UTC) from {username='kudu', principal='kudu/127.6.246.254@KRBTEST.COM'} at 127.0.0.1:49924
I20260925 22:38:44.287276 21648 tablet_replica.cc:330] stopping tablet replica
I20260925 22:38:44.287581 21648 raft_consensus.cc:2229] T 4aa187919d7f4bbc873f94938cf4144c P 593624e63fab46768718c022af35acde [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:38:44.287838 21648 raft_consensus.cc:2258] T 4aa187919d7f4bbc873f94938cf4144c P 593624e63fab46768718c022af35acde [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:38:44.288913 21648 ts_tablet_manager.cc:1905] T 4aa187919d7f4bbc873f94938cf4144c P 593624e63fab46768718c022af35acde: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:38:44.292656 21648 ts_tablet_manager.cc:1918] T 4aa187919d7f4bbc873f94938cf4144c P 593624e63fab46768718c022af35acde: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:38:44.292747 21648 log.cc:1196] T 4aa187919d7f4bbc873f94938cf4144c P 593624e63fab46768718c022af35acde: Deleting WAL directory at /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-0/wal/wals/4aa187919d7f4bbc873f94938cf4144c
I20260925 22:38:44.293077 21648 ts_tablet_manager.cc:1939] T 4aa187919d7f4bbc873f94938cf4144c P 593624e63fab46768718c022af35acde: Deleting consensus metadata
I20260925 22:38:44.294860 21436 catalog_manager.cc:4841] TS 593624e63fab46768718c022af35acde (127.6.246.193:33913): tablet 4aa187919d7f4bbc873f94938cf4144c (table test-table [id=4f762b8b27fc4d9fa1063685dc1fd8b2]) successfully deleted
W20260925 22:38:44.376071 21644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:38:44.376408 21644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:38:44.376543 21644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:38:44.381145 21644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:38:44.381228 21644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:38:44.381317 21644 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.194
I20260925 22:38:44.387248 21644 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:34883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.194.keytab
--principal=kudu/127.6.246.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.194:45893
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.246.194
--webserver_port=36019
--tserver_master_addrs=127.6.246.254:43017
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21644
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.194
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:38:44.388346 21644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:38:44.389408 21644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:38:44.399520 21653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:38:44.399606 21654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:38:44.399520 21656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:38:44.399722 21644 server_base.cc:1034] running on GCE node
I20260925 22:38:44.400187 21644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:38:44.400874 21644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:38:44.402072 21644 hybrid_clock.cc:648] HybridClock initialized: now 1790375924402062 us; error 32 us; skew 500 ppm
Sep 25 22:38:44 dist-test-slave-wjdr krb5kdc[20870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375924, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:38:44.404887 21644 init.cc:377] Logged in from keytab as kudu/127.6.246.194@KRBTEST.COM (short username kudu)
I20260925 22:38:44.405781 21644 webserver.cc:460] Webserver started at http://127.6.246.194:36019/ using document root <none> and password file <none>
I20260925 22:38:44.406342 21644 fs_manager.cc:359] Metadata directory not provided
I20260925 22:38:44.406446 21644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:38:44.410274 21644 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260925 22:38:44.412305 21663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:38:44.413607 21644 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:38:44.413739 21644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-1/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-1/wal
uuid: "6487020f6fdd432eaecee9f7c1c8beb1"
format_stamp: "Formatted at 2026-09-25 22:38:43 on dist-test-slave-wjdr"
server_key: "4642dbdfdadc8575fad1e4e9b7d083e8"
server_key_iv: "c5ee12b563ba3927f8ba289716028424"
server_key_version: "encryptionkey@0"
I20260925 22:38:44.414331 21644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:38:44.433511 21644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:38:44.436515 21644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:38:44.436715 21644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:38:44.437305 21644 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:38:44.438122 21644 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:38:44.438191 21644 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:38:44.438254 21644 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:38:44.438308 21644 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:38:44.447125 21644 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.194:45893
I20260925 22:38:44.447150 21776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.194:45893 every 8 connection(s)
I20260925 22:38:44.448628 21644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
I20260925 22:38:44.451474 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 21644
I20260925 22:38:44.451996 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.195:43353
--local_ip_for_outbound_sockets=127.6.246.195
--tserver_master_addrs=127.6.246.254:43017
--webserver_port=45167
--webserver_interface=127.6.246.195
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.195.keytab
--principal=kudu/127.6.246.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:34883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 22:38:44 dist-test-slave-wjdr krb5kdc[20870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375924, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.194@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
I20260925 22:38:44.467451 21641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:44.451322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.194:52375 (local address 127.6.246.254:43017)
0925 22:38:44.451525 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 22:38:44.451532 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:38:44.453070 (+ 1538us) server_negotiation.cc:407] Connection header received
0925 22:38:44.454552 (+ 1482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:38:44.454556 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:38:44.454629 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:38:44.454729 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:38:44.456518 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:44.457180 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:44.459243 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:44.459500 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:44.459594 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:44.463081 (+ 3487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:38:44.463085 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:38:44.463087 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:38:44.463117 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:38:44.465153 (+ 2036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:38:44.465725 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:38:44.465729 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:38:44.465731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:38:44.465824 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:38:44.466179 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:38:44.466184 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:38:44.466186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:38:44.466402 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:38:44.466509 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 22:38:44.467091 (+ 582us) server_negotiation.cc:299] Negotiation successful
0925 22:38:44.467254 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 22:38:44.468515 21779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:44.451792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43017 (local address 127.6.246.194:52375)
0925 22:38:44.452814 (+ 1022us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:44.452885 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0925 22:38:44.454289 (+ 1404us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:38:44.454929 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:38:44.454937 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:38:44.455480 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:38:44.456288 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:38:44.456311 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:38:44.457331 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:38:44.457337 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:38:44.459098 (+ 1761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:38:44.459114 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:38:44.459688 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:38:44.459695 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:38:44.459971 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:44.460842 (+ 871us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:38:44.460873 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:38:44.462873 (+ 2000us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:38:44.465290 (+ 2417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:38:44.465297 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:38:44.465317 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:38:44.465611 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:38:44.465921 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:38:44.465924 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:38:44.465926 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:38:44.466086 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:38:44.466513 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:38:44.466523 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:38:44.466830 (+ 307us) client_negotiation.cc:769] Sending connection context
0925 22:38:44.467043 (+ 213us) client_negotiation.cc:240] Negotiation successful
0925 22:38:44.467452 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":714,"thread_start_us":164,"threads_started":1}
I20260925 22:38:44.469918 21777 heartbeater.cc:347] Connected to a master server at 127.6.246.254:43017
I20260925 22:38:44.470232 21777 heartbeater.cc:464] Registering TS with master...
I20260925 22:38:44.470820 21777 heartbeater.cc:511] Master 127.6.246.254:43017 requested a full tablet report, sending...
I20260925 22:38:44.472072 21451 ts_manager.cc:194] Registered new tserver with Master: 6487020f6fdd432eaecee9f7c1c8beb1 (127.6.246.194:45893)
I20260925 22:38:44.472951 21451 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.246.194@KRBTEST.COM'} at 127.6.246.194:52375
W20260925 22:38:44.587714 21780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:38:44.588076 21780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:38:44.588125 21780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:38:44.592489 21780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:38:44.592562 21780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:38:44.592629 21780 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.195
I20260925 22:38:44.598680 21780 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:34883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.195.keytab
--principal=kudu/127.6.246.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.195:43353
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.246.195
--webserver_port=45167
--tserver_master_addrs=127.6.246.254:43017
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.195
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:38:44.600075 21780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:38:44.601300 21780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:38:44.612519 21786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:38:44.612521 21788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:38:44.612596 21785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:38:44.613003 21780 server_base.cc:1034] running on GCE node
I20260925 22:38:44.613457 21780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:38:44.614305 21780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:38:44.615502 21780 hybrid_clock.cc:648] HybridClock initialized: now 1790375924615476 us; error 40 us; skew 500 ppm
Sep 25 22:38:44 dist-test-slave-wjdr krb5kdc[20870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375924, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:38:44.618877 21780 init.cc:377] Logged in from keytab as kudu/127.6.246.195@KRBTEST.COM (short username kudu)
I20260925 22:38:44.619905 21780 webserver.cc:460] Webserver started at http://127.6.246.195:45167/ using document root <none> and password file <none>
I20260925 22:38:44.620643 21780 fs_manager.cc:359] Metadata directory not provided
I20260925 22:38:44.620736 21780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:38:44.625269 21780 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 22:38:44.627573 21795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:38:44.629091 21780 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:38:44.629233 21780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-2/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-2/wal
uuid: "2f130a10f1e64d5e8439fa19988dfee5"
format_stamp: "Formatted at 2026-09-25 22:38:43 on dist-test-slave-wjdr"
server_key: "07c03525d4e9c4e6ce3969574621eb3c"
server_key_iv: "2235cc6693edd0c629bd98ee65b2c830"
server_key_version: "encryptionkey@0"
I20260925 22:38:44.629871 21780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:38:44.649919 21780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:38:44.653590 21780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:38:44.653864 21780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:38:44.654603 21780 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:38:44.655581 21780 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:38:44.655629 21780 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:38:44.655701 21780 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:38:44.655737 21780 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:38:44.666347 21780 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.195:43353
I20260925 22:38:44.666366 21908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.195:43353 every 8 connection(s)
I20260925 22:38:44.667565 21780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
I20260925 22:38:44.671380 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 21780
Sep 25 22:38:44 dist-test-slave-wjdr krb5kdc[20870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375924, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.195@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
I20260925 22:38:44.681069 21913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:44.672230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56148 (local address 127.6.246.254:43017)
0925 22:38:44.672651 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0925 22:38:44.672656 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:38:44.672800 (+ 144us) server_negotiation.cc:407] Connection header received
0925 22:38:44.672951 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:38:44.672954 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:38:44.673001 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:38:44.673103 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:38:44.673915 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:44.674545 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:44.675413 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:44.675652 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:44.675759 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:44.677427 (+ 1668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:38:44.677431 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:38:44.677434 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:38:44.677463 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:38:44.679164 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:38:44.679657 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:38:44.679664 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:38:44.679666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:38:44.679757 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:38:44.680030 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:38:44.680033 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:38:44.680035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:38:44.680231 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:38:44.680504 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0925 22:38:44.680639 (+ 135us) server_negotiation.cc:299] Negotiation successful
0925 22:38:44.680764 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":119,"threads_started":1}
I20260925 22:38:44.682354 21641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:44.669702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.195:52869 (local address 127.6.246.254:43017)
0925 22:38:44.669872 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 22:38:44.669876 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:38:44.671127 (+ 1251us) server_negotiation.cc:407] Connection header received
0925 22:38:44.672503 (+ 1376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:38:44.672507 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:38:44.672573 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:38:44.672682 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:38:44.674080 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:44.674581 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:44.675772 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:44.675996 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:44.676099 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:44.678608 (+ 2509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:38:44.678612 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:38:44.678615 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:38:44.678661 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:38:44.679978 (+ 1317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:38:44.680528 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:38:44.680531 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:38:44.680533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:38:44.680638 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:38:44.681057 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:38:44.681060 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:38:44.681062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:38:44.681327 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:38:44.681434 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 22:38:44.682086 (+ 652us) server_negotiation.cc:299] Negotiation successful
0925 22:38:44.682190 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 22:38:44.683316 21911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:44.670160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43017 (local address 127.6.246.195:52869)
0925 22:38:44.670924 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:44.670969 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 22:38:44.672140 (+ 1171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:38:44.672891 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:38:44.672898 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:38:44.673305 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:38:44.673919 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:38:44.673934 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:38:44.674705 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:38:44.674709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:38:44.675636 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:38:44.675645 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:38:44.676151 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:38:44.676156 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:38:44.676331 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:44.676912 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:38:44.676929 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:38:44.678445 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:38:44.680103 (+ 1658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:38:44.680111 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:38:44.680126 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:38:44.680393 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:38:44.680766 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:38:44.680770 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:38:44.680774 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:38:44.680957 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:38:44.681434 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:38:44.681442 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:38:44.681785 (+ 343us) client_negotiation.cc:769] Sending connection context
0925 22:38:44.682014 (+ 229us) client_negotiation.cc:240] Negotiation successful
0925 22:38:44.682376 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":130,"threads_started":1}
I20260925 22:38:44.684646 21909 heartbeater.cc:347] Connected to a master server at 127.6.246.254:43017
I20260925 22:38:44.684921 21909 heartbeater.cc:464] Registering TS with master...
I20260925 22:38:44.685530 21909 heartbeater.cc:511] Master 127.6.246.254:43017 requested a full tablet report, sending...
I20260925 22:38:44.686725 21451 ts_manager.cc:194] Registered new tserver with Master: 2f130a10f1e64d5e8439fa19988dfee5 (127.6.246.195:43353)
I20260925 22:38:44.687399 21451 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.246.195@KRBTEST.COM'} at 127.6.246.195:52869
I20260925 22:38:44.694936 7131 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:38:44.699945 21641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:44.695773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56164 (local address 127.6.246.254:43017)
0925 22:38:44.695913 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 22:38:44.695918 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:38:44.696138 (+ 220us) server_negotiation.cc:407] Connection header received
0925 22:38:44.696269 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:38:44.696282 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:38:44.696401 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:38:44.696646 (+ 245us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:38:44.696941 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:44.697556 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:44.698406 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:44.698620 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:44.698697 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:44.699013 (+ 316us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:38:44.699501 (+ 488us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:38:44.699612 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 22:38:44.699731 (+ 119us) server_negotiation.cc:299] Negotiation successful
0925 22:38:44.699777 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 22:38:44.700783 21451 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:56164:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:38:44.704670 21451 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:38:44.714119 21573 tablet_service.cc:1467] Processing CreateTablet for tablet a209003a4bdb4c28b446d2168ead92f2 (DEFAULT_TABLE table=test-table [id=c6d8c4d4219f40389bb1b50126d39829]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:38:44.715001 21573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a209003a4bdb4c28b446d2168ead92f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:38:44.718436 21646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:44.712647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.195:43353 (local address 127.0.0.1:55254)
0925 22:38:44.712834 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:44.712850 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:38:44.712954 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:38:44.714171 (+ 1217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:38:44.714174 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:38:44.714194 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:38:44.714339 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:38:44.714349 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:38:44.715676 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:38:44.715680 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:38:44.716860 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:38:44.716870 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:38:44.717973 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:38:44.717976 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:38:44.718076 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:44.718123 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 22:38:44.718193 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 22:38:44.718288 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107}
I20260925 22:38:44.718734 21531 tablet_bootstrap.cc:492] T a209003a4bdb4c28b446d2168ead92f2 P 593624e63fab46768718c022af35acde: Bootstrap starting.
I20260925 22:38:44.718469 21919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:44.712857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55254 (local address 127.6.246.195:43353)
0925 22:38:44.713490 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0925 22:38:44.713506 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 22:38:44.713539 (+ 33us) server_negotiation.cc:407] Connection header received
0925 22:38:44.713626 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:38:44.713630 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:38:44.713918 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:38:44.714189 (+ 271us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:38:44.714617 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:44.715544 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:44.716999 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:44.717841 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:44.717958 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:44.718167 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0925 22:38:44.718229 (+ 62us) server_negotiation.cc:299] Negotiation successful
0925 22:38:44.718330 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":202,"threads_started":1}
I20260925 22:38:44.718950 21918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:44.712841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.194:45893 (local address 127.0.0.1:59168)
0925 22:38:44.713239 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:44.713253 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:38:44.713358 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:38:44.714527 (+ 1169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:38:44.714530 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:38:44.714545 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:38:44.714672 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:38:44.714678 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:38:44.716169 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:38:44.716172 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:38:44.717327 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:38:44.717336 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:38:44.718533 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:38:44.718537 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:38:44.718656 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:44.718681 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 22:38:44.718734 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 22:38:44.718823 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":163,"threads_started":1}
I20260925 22:38:44.719604 21920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:44.713185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59168 (local address 127.6.246.194:45893)
0925 22:38:44.713616 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0925 22:38:44.713629 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 22:38:44.713652 (+ 23us) server_negotiation.cc:407] Connection header received
0925 22:38:44.713735 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:38:44.713740 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:38:44.713995 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:38:44.714813 (+ 818us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:38:44.714997 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:44.716020 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:44.717505 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:44.718394 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:44.719221 (+ 827us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:44.719331 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 22:38:44.719371 (+ 40us) server_negotiation.cc:299] Negotiation successful
0925 22:38:44.719445 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":132,"threads_started":1}
I20260925 22:38:44.719866 21531 tablet_bootstrap.cc:654] T a209003a4bdb4c28b446d2168ead92f2 P 593624e63fab46768718c022af35acde: Neither blocks nor log segments found. Creating new log.
I20260925 22:38:44.721457 21531 tablet_bootstrap.cc:492] T a209003a4bdb4c28b446d2168ead92f2 P 593624e63fab46768718c022af35acde: No bootstrap required, opened a new log
I20260925 22:38:44.721570 21531 ts_tablet_manager.cc:1397] T a209003a4bdb4c28b446d2168ead92f2 P 593624e63fab46768718c022af35acde: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 22:38:44.722088 21531 raft_consensus.cc:348] T a209003a4bdb4c28b446d2168ead92f2 P 593624e63fab46768718c022af35acde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f130a10f1e64d5e8439fa19988dfee5" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43353 } } peers { permanent_uuid: "6487020f6fdd432eaecee9f7c1c8beb1" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 45893 } } peers { permanent_uuid: "593624e63fab46768718c022af35acde" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 33913 } }
I20260925 22:38:44.722231 21531 raft_consensus.cc:374] T a209003a4bdb4c28b446d2168ead92f2 P 593624e63fab46768718c022af35acde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:38:44.722278 21531 raft_consensus.cc:729] T a209003a4bdb4c28b446d2168ead92f2 P 593624e63fab46768718c022af35acde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 593624e63fab46768718c022af35acde, State: Initialized, Role: FOLLOWER
I20260925 22:38:44.722420 21531 consensus_queue.cc:260] T a209003a4bdb4c28b446d2168ead92f2 P 593624e63fab46768718c022af35acde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f130a10f1e64d5e8439fa19988dfee5" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43353 } } peers { permanent_uuid: "6487020f6fdd432eaecee9f7c1c8beb1" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 45893 } } peers { permanent_uuid: "593624e63fab46768718c022af35acde" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 33913 } }
I20260925 22:38:44.722811 21531 ts_tablet_manager.cc:1428] T a209003a4bdb4c28b446d2168ead92f2 P 593624e63fab46768718c022af35acde: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260925 22:38:44.723100 21711 tablet_service.cc:1467] Processing CreateTablet for tablet a209003a4bdb4c28b446d2168ead92f2 (DEFAULT_TABLE table=test-table [id=c6d8c4d4219f40389bb1b50126d39829]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:38:44.723094 21843 tablet_service.cc:1467] Processing CreateTablet for tablet a209003a4bdb4c28b446d2168ead92f2 (DEFAULT_TABLE table=test-table [id=c6d8c4d4219f40389bb1b50126d39829]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:38:44.724360 21711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a209003a4bdb4c28b446d2168ead92f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:38:44.724378 21843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a209003a4bdb4c28b446d2168ead92f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:38:44.732255 21923 tablet_bootstrap.cc:492] T a209003a4bdb4c28b446d2168ead92f2 P 6487020f6fdd432eaecee9f7c1c8beb1: Bootstrap starting.
I20260925 22:38:44.732255 21922 tablet_bootstrap.cc:492] T a209003a4bdb4c28b446d2168ead92f2 P 2f130a10f1e64d5e8439fa19988dfee5: Bootstrap starting.
I20260925 22:38:44.735142 21922 tablet_bootstrap.cc:654] T a209003a4bdb4c28b446d2168ead92f2 P 2f130a10f1e64d5e8439fa19988dfee5: Neither blocks nor log segments found. Creating new log.
I20260925 22:38:44.735142 21923 tablet_bootstrap.cc:654] T a209003a4bdb4c28b446d2168ead92f2 P 6487020f6fdd432eaecee9f7c1c8beb1: Neither blocks nor log segments found. Creating new log.
I20260925 22:38:44.736081 21923 log.cc:824] T a209003a4bdb4c28b446d2168ead92f2 P 6487020f6fdd432eaecee9f7c1c8beb1: Log is configured to *not* fsync() on all Append() calls
I20260925 22:38:44.736081 21922 log.cc:824] T a209003a4bdb4c28b446d2168ead92f2 P 2f130a10f1e64d5e8439fa19988dfee5: Log is configured to *not* fsync() on all Append() calls
I20260925 22:38:44.738085 21923 tablet_bootstrap.cc:492] T a209003a4bdb4c28b446d2168ead92f2 P 6487020f6fdd432eaecee9f7c1c8beb1: No bootstrap required, opened a new log
I20260925 22:38:44.738099 21922 tablet_bootstrap.cc:492] T a209003a4bdb4c28b446d2168ead92f2 P 2f130a10f1e64d5e8439fa19988dfee5: No bootstrap required, opened a new log
I20260925 22:38:44.738269 21923 ts_tablet_manager.cc:1397] T a209003a4bdb4c28b446d2168ead92f2 P 6487020f6fdd432eaecee9f7c1c8beb1: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 22:38:44.738281 21922 ts_tablet_manager.cc:1397] T a209003a4bdb4c28b446d2168ead92f2 P 2f130a10f1e64d5e8439fa19988dfee5: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 22:38:44.741406 21923 raft_consensus.cc:348] T a209003a4bdb4c28b446d2168ead92f2 P 6487020f6fdd432eaecee9f7c1c8beb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f130a10f1e64d5e8439fa19988dfee5" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43353 } } peers { permanent_uuid: "6487020f6fdd432eaecee9f7c1c8beb1" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 45893 } } peers { permanent_uuid: "593624e63fab46768718c022af35acde" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 33913 } }
I20260925 22:38:44.741405 21922 raft_consensus.cc:348] T a209003a4bdb4c28b446d2168ead92f2 P 2f130a10f1e64d5e8439fa19988dfee5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f130a10f1e64d5e8439fa19988dfee5" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43353 } } peers { permanent_uuid: "6487020f6fdd432eaecee9f7c1c8beb1" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 45893 } } peers { permanent_uuid: "593624e63fab46768718c022af35acde" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 33913 } }
I20260925 22:38:44.741670 21923 raft_consensus.cc:374] T a209003a4bdb4c28b446d2168ead92f2 P 6487020f6fdd432eaecee9f7c1c8beb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:38:44.741707 21923 raft_consensus.cc:729] T a209003a4bdb4c28b446d2168ead92f2 P 6487020f6fdd432eaecee9f7c1c8beb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6487020f6fdd432eaecee9f7c1c8beb1, State: Initialized, Role: FOLLOWER
I20260925 22:38:44.741698 21922 raft_consensus.cc:374] T a209003a4bdb4c28b446d2168ead92f2 P 2f130a10f1e64d5e8439fa19988dfee5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:38:44.741760 21922 raft_consensus.cc:729] T a209003a4bdb4c28b446d2168ead92f2 P 2f130a10f1e64d5e8439fa19988dfee5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f130a10f1e64d5e8439fa19988dfee5, State: Initialized, Role: FOLLOWER
I20260925 22:38:44.742260 21923 consensus_queue.cc:260] T a209003a4bdb4c28b446d2168ead92f2 P 6487020f6fdd432eaecee9f7c1c8beb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f130a10f1e64d5e8439fa19988dfee5" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43353 } } peers { permanent_uuid: "6487020f6fdd432eaecee9f7c1c8beb1" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 45893 } } peers { permanent_uuid: "593624e63fab46768718c022af35acde" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 33913 } }
I20260925 22:38:44.742259 21922 consensus_queue.cc:260] T a209003a4bdb4c28b446d2168ead92f2 P 2f130a10f1e64d5e8439fa19988dfee5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f130a10f1e64d5e8439fa19988dfee5" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43353 } } peers { permanent_uuid: "6487020f6fdd432eaecee9f7c1c8beb1" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 45893 } } peers { permanent_uuid: "593624e63fab46768718c022af35acde" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 33913 } }
I20260925 22:38:44.743036 21909 heartbeater.cc:503] Master 127.6.246.254:43017 was elected leader, sending a full tablet report...
I20260925 22:38:44.743256 21777 heartbeater.cc:503] Master 127.6.246.254:43017 was elected leader, sending a full tablet report...
I20260925 22:38:44.743426 21923 ts_tablet_manager.cc:1428] T a209003a4bdb4c28b446d2168ead92f2 P 6487020f6fdd432eaecee9f7c1c8beb1: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260925 22:38:44.743426 21922 ts_tablet_manager.cc:1428] T a209003a4bdb4c28b446d2168ead92f2 P 2f130a10f1e64d5e8439fa19988dfee5: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260925 22:38:44.746410 21640 tablet.cc:2369] T a209003a4bdb4c28b446d2168ead92f2 P 593624e63fab46768718c022af35acde: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 22:38:44.918800 21910 tablet.cc:2369] T a209003a4bdb4c28b446d2168ead92f2 P 2f130a10f1e64d5e8439fa19988dfee5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 22:38:44.950196 21778 tablet.cc:2369] T a209003a4bdb4c28b446d2168ead92f2 P 6487020f6fdd432eaecee9f7c1c8beb1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:38:44.978226 21926 raft_consensus.cc:482] T a209003a4bdb4c28b446d2168ead92f2 P 2f130a10f1e64d5e8439fa19988dfee5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:38:44.978566 21926 raft_consensus.cc:504] T a209003a4bdb4c28b446d2168ead92f2 P 2f130a10f1e64d5e8439fa19988dfee5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f130a10f1e64d5e8439fa19988dfee5" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43353 } } peers { permanent_uuid: "6487020f6fdd432eaecee9f7c1c8beb1" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 45893 } } peers { permanent_uuid: "593624e63fab46768718c022af35acde" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 33913 } }
I20260925 22:38:44.980259 21926 leader_election.cc:288] T a209003a4bdb4c28b446d2168ead92f2 P 2f130a10f1e64d5e8439fa19988dfee5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6487020f6fdd432eaecee9f7c1c8beb1 (127.6.246.194:45893), 593624e63fab46768718c022af35acde (127.6.246.193:33913)
I20260925 22:38:44.984820 21911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:44.980615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.194:45893 (local address 127.6.246.195:51149)
0925 22:38:44.980871 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:44.980890 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 22:38:44.981019 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:38:44.981363 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:38:44.981366 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:38:44.981385 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:38:44.981585 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:38:44.981594 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:38:44.982606 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:38:44.982609 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:38:44.983514 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:38:44.983523 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:38:44.984270 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:38:44.984273 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:38:44.984388 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:44.984417 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 22:38:44.984539 (+ 122us) client_negotiation.cc:240] Negotiation successful
0925 22:38:44.984654 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146}
I20260925 22:38:44.984819 21920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:44.980708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.195:51149 (local address 127.6.246.194:45893)
0925 22:38:44.980871 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 22:38:44.980876 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:38:44.980997 (+ 121us) server_negotiation.cc:407] Connection header received
0925 22:38:44.981171 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:38:44.981177 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:38:44.981251 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:38:44.981341 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:38:44.981748 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:44.982462 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:44.983641 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:44.984160 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:44.984259 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:44.984308 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 22:38:44.984558 (+ 250us) server_negotiation.cc:299] Negotiation successful
0925 22:38:44.984654 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 22:38:44.985369 21928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:44.980640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.193:33913 (local address 127.6.246.195:53583)
0925 22:38:44.981144 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:44.981159 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:38:44.981253 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:38:44.981748 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:38:44.981751 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:38:44.981770 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:38:44.981887 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:38:44.981894 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:38:44.982854 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:38:44.982856 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:38:44.983795 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:38:44.983804 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:38:44.984930 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:38:44.984934 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:38:44.985035 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:44.985063 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 22:38:44.985154 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 22:38:44.985225 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"mutex_wait_us":4,"thread_start_us":158,"threads_started":1}
I20260925 22:38:44.985395 21929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:44.980777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.195:53583 (local address 127.6.246.193:33913)
0925 22:38:44.981207 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0925 22:38:44.981211 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:38:44.981225 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:38:44.981377 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:38:44.981381 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:38:44.981456 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:38:44.981595 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:38:44.982041 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:44.982732 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:44.983929 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:44.984561 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:44.984672 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:44.984846 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0925 22:38:44.985174 (+ 328us) server_negotiation.cc:299] Negotiation successful
0925 22:38:44.985255 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":130,"threads_started":1}
I20260925 22:38:44.985687 21731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a209003a4bdb4c28b446d2168ead92f2" candidate_uuid: "2f130a10f1e64d5e8439fa19988dfee5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6487020f6fdd432eaecee9f7c1c8beb1" is_pre_election: true
I20260925 22:38:44.985956 21593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a209003a4bdb4c28b446d2168ead92f2" candidate_uuid: "2f130a10f1e64d5e8439fa19988dfee5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "593624e63fab46768718c022af35acde" is_pre_election: true
I20260925 22:38:44.986083 21731 raft_consensus.cc:2454] T a209003a4bdb4c28b446d2168ead92f2 P 6487020f6fdd432eaecee9f7c1c8beb1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f130a10f1e64d5e8439fa19988dfee5 in term 0.
I20260925 22:38:44.986276 21593 raft_consensus.cc:2454] T a209003a4bdb4c28b446d2168ead92f2 P 593624e63fab46768718c022af35acde [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f130a10f1e64d5e8439fa19988dfee5 in term 0.
I20260925 22:38:44.986645 21798 leader_election.cc:302] T a209003a4bdb4c28b446d2168ead92f2 P 2f130a10f1e64d5e8439fa19988dfee5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f130a10f1e64d5e8439fa19988dfee5, 6487020f6fdd432eaecee9f7c1c8beb1; no voters:
I20260925 22:38:44.986951 21926 raft_consensus.cc:2781] T a209003a4bdb4c28b446d2168ead92f2 P 2f130a10f1e64d5e8439fa19988dfee5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:38:44.987039 21926 raft_consensus.cc:482] T a209003a4bdb4c28b446d2168ead92f2 P 2f130a10f1e64d5e8439fa19988dfee5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:38:44.987092 21926 raft_consensus.cc:3037] T a209003a4bdb4c28b446d2168ead92f2 P 2f130a10f1e64d5e8439fa19988dfee5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:38:44.988348 21926 raft_consensus.cc:504] T a209003a4bdb4c28b446d2168ead92f2 P 2f130a10f1e64d5e8439fa19988dfee5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f130a10f1e64d5e8439fa19988dfee5" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43353 } } peers { permanent_uuid: "6487020f6fdd432eaecee9f7c1c8beb1" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 45893 } } peers { permanent_uuid: "593624e63fab46768718c022af35acde" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 33913 } }
I20260925 22:38:44.988796 21926 leader_election.cc:288] T a209003a4bdb4c28b446d2168ead92f2 P 2f130a10f1e64d5e8439fa19988dfee5 [CANDIDATE]: Term 1 election: Requested vote from peers 6487020f6fdd432eaecee9f7c1c8beb1 (127.6.246.194:45893), 593624e63fab46768718c022af35acde (127.6.246.193:33913)
I20260925 22:38:44.989176 21731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a209003a4bdb4c28b446d2168ead92f2" candidate_uuid: "2f130a10f1e64d5e8439fa19988dfee5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6487020f6fdd432eaecee9f7c1c8beb1"
I20260925 22:38:44.989176 21593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a209003a4bdb4c28b446d2168ead92f2" candidate_uuid: "2f130a10f1e64d5e8439fa19988dfee5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "593624e63fab46768718c022af35acde"
I20260925 22:38:44.989311 21593 raft_consensus.cc:3037] T a209003a4bdb4c28b446d2168ead92f2 P 593624e63fab46768718c022af35acde [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:38:44.989311 21731 raft_consensus.cc:3037] T a209003a4bdb4c28b446d2168ead92f2 P 6487020f6fdd432eaecee9f7c1c8beb1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:38:44.990743 21593 raft_consensus.cc:2454] T a209003a4bdb4c28b446d2168ead92f2 P 593624e63fab46768718c022af35acde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f130a10f1e64d5e8439fa19988dfee5 in term 1.
I20260925 22:38:44.990743 21731 raft_consensus.cc:2454] T a209003a4bdb4c28b446d2168ead92f2 P 6487020f6fdd432eaecee9f7c1c8beb1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f130a10f1e64d5e8439fa19988dfee5 in term 1.
I20260925 22:38:44.991079 21797 leader_election.cc:302] T a209003a4bdb4c28b446d2168ead92f2 P 2f130a10f1e64d5e8439fa19988dfee5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f130a10f1e64d5e8439fa19988dfee5, 593624e63fab46768718c022af35acde; no voters:
I20260925 22:38:44.991255 21926 raft_consensus.cc:2781] T a209003a4bdb4c28b446d2168ead92f2 P 2f130a10f1e64d5e8439fa19988dfee5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:38:44.991539 21926 raft_consensus.cc:686] T a209003a4bdb4c28b446d2168ead92f2 P 2f130a10f1e64d5e8439fa19988dfee5 [term 1 LEADER]: Becoming Leader. State: Replica: 2f130a10f1e64d5e8439fa19988dfee5, State: Running, Role: LEADER
I20260925 22:38:44.991940 21926 consensus_queue.cc:237] T a209003a4bdb4c28b446d2168ead92f2 P 2f130a10f1e64d5e8439fa19988dfee5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f130a10f1e64d5e8439fa19988dfee5" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43353 } } peers { permanent_uuid: "6487020f6fdd432eaecee9f7c1c8beb1" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 45893 } } peers { permanent_uuid: "593624e63fab46768718c022af35acde" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 33913 } }
I20260925 22:38:44.995843 21451 catalog_manager.cc:5495] T a209003a4bdb4c28b446d2168ead92f2 P 2f130a10f1e64d5e8439fa19988dfee5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f130a10f1e64d5e8439fa19988dfee5 (127.6.246.195). New cstate: current_term: 1 leader_uuid: "2f130a10f1e64d5e8439fa19988dfee5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f130a10f1e64d5e8439fa19988dfee5" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6487020f6fdd432eaecee9f7c1c8beb1" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 45893 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "593624e63fab46768718c022af35acde" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 33913 } health_report { overall_health: UNKNOWN } } }
I20260925 22:38:45.015221 21919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:45.011360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55262 (local address 127.6.246.195:43353)
0925 22:38:45.011501 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 22:38:45.011506 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:38:45.011522 (+ 16us) server_negotiation.cc:407] Connection header received
0925 22:38:45.011663 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:38:45.011665 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:38:45.011732 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:38:45.011859 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:38:45.012132 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:45.012746 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:45.013613 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:45.013875 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:45.013983 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:45.014318 (+ 335us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:38:45.014796 (+ 478us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:38:45.014934 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 22:38:45.014975 (+ 41us) server_negotiation.cc:299] Negotiation successful
0925 22:38:45.015039 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 22:38:45.022758 21731 raft_consensus.cc:1261] T a209003a4bdb4c28b446d2168ead92f2 P 6487020f6fdd432eaecee9f7c1c8beb1 [term 1 FOLLOWER]: Refusing update from remote peer 2f130a10f1e64d5e8439fa19988dfee5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:38:45.022755 21593 raft_consensus.cc:1261] T a209003a4bdb4c28b446d2168ead92f2 P 593624e63fab46768718c022af35acde [term 1 FOLLOWER]: Refusing update from remote peer 2f130a10f1e64d5e8439fa19988dfee5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:38:45.023720 21930 consensus_queue.cc:1035] T a209003a4bdb4c28b446d2168ead92f2 P 2f130a10f1e64d5e8439fa19988dfee5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "593624e63fab46768718c022af35acde" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:38:45.023931 21926 consensus_queue.cc:1035] T a209003a4bdb4c28b446d2168ead92f2 P 2f130a10f1e64d5e8439fa19988dfee5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6487020f6fdd432eaecee9f7c1c8beb1" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 45893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:38:45.031237 21934 mvcc.cc:205] Tried to move back new op lower bound from 7333379788883472384 to 7333379788770189312. Current Snapshot: MvccSnapshot[applied={T|T < 7333379788883472384}]
I20260925 22:38:45.042425 21450 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:56164:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:38:45.042634 21450 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:56164:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:38:45.045624 21450 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 761938f0951e48a0910db98fde88b05f: Sending DeleteTablet for 3 replicas of tablet a209003a4bdb4c28b446d2168ead92f2
I20260925 22:38:45.046588 21573 tablet_service.cc:1514] Processing DeleteTablet for tablet a209003a4bdb4c28b446d2168ead92f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:38:45 UTC) from {username='kudu', principal='kudu/127.6.246.254@KRBTEST.COM'} at 127.0.0.1:49924
I20260925 22:38:45.046593 21843 tablet_service.cc:1514] Processing DeleteTablet for tablet a209003a4bdb4c28b446d2168ead92f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:38:45 UTC) from {username='kudu', principal='kudu/127.6.246.254@KRBTEST.COM'} at 127.0.0.1:55254
I20260925 22:38:45.046592 21711 tablet_service.cc:1514] Processing DeleteTablet for tablet a209003a4bdb4c28b446d2168ead92f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:38:45 UTC) from {username='kudu', principal='kudu/127.6.246.254@KRBTEST.COM'} at 127.0.0.1:59168
I20260925 22:38:45.047152 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 21489
I20260925 22:38:45.047184 21944 tablet_replica.cc:330] T a209003a4bdb4c28b446d2168ead92f2 P 2f130a10f1e64d5e8439fa19988dfee5: stopping tablet replica
I20260925 22:38:45.048022 21945 tablet_replica.cc:330] T a209003a4bdb4c28b446d2168ead92f2 P 6487020f6fdd432eaecee9f7c1c8beb1: stopping tablet replica
I20260925 22:38:45.048556 21945 raft_consensus.cc:2229] T a209003a4bdb4c28b446d2168ead92f2 P 6487020f6fdd432eaecee9f7c1c8beb1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:38:45.048723 21944 raft_consensus.cc:2229] T a209003a4bdb4c28b446d2168ead92f2 P 2f130a10f1e64d5e8439fa19988dfee5 [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:38:45.049089 21945 raft_consensus.cc:2258] T a209003a4bdb4c28b446d2168ead92f2 P 6487020f6fdd432eaecee9f7c1c8beb1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:38:45.049212 21944 raft_consensus.cc:2258] T a209003a4bdb4c28b446d2168ead92f2 P 2f130a10f1e64d5e8439fa19988dfee5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:38:45.051123 21944 ts_tablet_manager.cc:1905] T a209003a4bdb4c28b446d2168ead92f2 P 2f130a10f1e64d5e8439fa19988dfee5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:38:45.051123 21945 ts_tablet_manager.cc:1905] T a209003a4bdb4c28b446d2168ead92f2 P 6487020f6fdd432eaecee9f7c1c8beb1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:38:45.054924 21944 ts_tablet_manager.cc:1918] T a209003a4bdb4c28b446d2168ead92f2 P 2f130a10f1e64d5e8439fa19988dfee5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:38:45.055047 21944 log.cc:1196] T a209003a4bdb4c28b446d2168ead92f2 P 2f130a10f1e64d5e8439fa19988dfee5: Deleting WAL directory at /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-2/wal/wals/a209003a4bdb4c28b446d2168ead92f2
I20260925 22:38:45.055258 21945 ts_tablet_manager.cc:1918] T a209003a4bdb4c28b446d2168ead92f2 P 6487020f6fdd432eaecee9f7c1c8beb1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:38:45.055423 21944 ts_tablet_manager.cc:1939] T a209003a4bdb4c28b446d2168ead92f2 P 2f130a10f1e64d5e8439fa19988dfee5: Deleting consensus metadata
I20260925 22:38:45.055431 21945 log.cc:1196] T a209003a4bdb4c28b446d2168ead92f2 P 6487020f6fdd432eaecee9f7c1c8beb1: Deleting WAL directory at /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-1/wal/wals/a209003a4bdb4c28b446d2168ead92f2
I20260925 22:38:45.055879 21945 ts_tablet_manager.cc:1939] T a209003a4bdb4c28b446d2168ead92f2 P 6487020f6fdd432eaecee9f7c1c8beb1: Deleting consensus metadata
W20260925 22:38:45.057122 21436 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.246.193:33913 (error 108)
I20260925 22:38:45.057307 21437 catalog_manager.cc:4841] TS 6487020f6fdd432eaecee9f7c1c8beb1 (127.6.246.194:45893): tablet a209003a4bdb4c28b446d2168ead92f2 (table test-table [id=c6d8c4d4219f40389bb1b50126d39829]) successfully deleted
I20260925 22:38:45.057688 21436 catalog_manager.cc:4841] TS 2f130a10f1e64d5e8439fa19988dfee5 (127.6.246.195:43353): tablet a209003a4bdb4c28b446d2168ead92f2 (table test-table [id=c6d8c4d4219f40389bb1b50126d39829]) successfully deleted
I20260925 22:38:45.057827 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 21644
I20260925 22:38:45.059725 21918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:45.059278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.193:33913 (local address 127.0.0.1:49930)
0925 22:38:45.059500 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:45.059589 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.193:33913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
W20260925 22:38:45.059975 21436 catalog_manager.cc:4584] TS 593624e63fab46768718c022af35acde (127.6.246.193:33913): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a209003a4bdb4c28b446d2168ead92f2: Network error: Client connection negotiation failed: client connection to 127.6.246.193:33913: connect: Connection refused (error 111)
I20260925 22:38:45.065798 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 21780
I20260925 22:38:45.071791 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 21418
I20260925 22:38:45.079972 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:43017
--webserver_interface=127.6.246.254
--webserver_port=33505
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab
--principal=kudu/127.6.246.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:34883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.246.254:43017
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:38:45.209682 21947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:38:45.210017 21947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:38:45.210060 21947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:38:45.214329 21947 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:38:45.214407 21947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:38:45.214429 21947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:38:45.214447 21947 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:38:45.214463 21947 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:38:45.220533 21947 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:34883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.246.254:43017
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab
--principal=kudu/127.6.246.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:43017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.246.254
--webserver_port=33505
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21947
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:38:45.221834 21947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:38:45.223191 21947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:38:45.232992 21953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:38:45.233127 21947 server_base.cc:1034] running on GCE node
W20260925 22:38:45.233003 21952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:38:45.232992 21955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:38:45.234019 21947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:38:45.235328 21947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:38:45.236553 21947 hybrid_clock.cc:648] HybridClock initialized: now 1790375925236530 us; error 40 us; skew 500 ppm
Sep 25 22:38:45 dist-test-slave-wjdr krb5kdc[20870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375925, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:38:45.239967 21947 init.cc:377] Logged in from keytab as kudu/127.6.246.254@KRBTEST.COM (short username kudu)
I20260925 22:38:45.241199 21947 webserver.cc:460] Webserver started at http://127.6.246.254:33505/ using document root <none> and password file <none>
I20260925 22:38:45.241905 21947 fs_manager.cc:359] Metadata directory not provided
I20260925 22:38:45.241972 21947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:38:45.246783 21947 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:38:45.249352 21962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:38:45.250967 21947 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:38:45.251144 21947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/master-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/master-0/wal
uuid: "761938f0951e48a0910db98fde88b05f"
format_stamp: "Formatted at 2026-09-25 22:38:42 on dist-test-slave-wjdr"
server_key: "18768535b2555fb849ed97cf45aebc3c"
server_key_iv: "5804c09b287daae23dfb0a1be6d786f4"
server_key_version: "encryptionkey@0"
I20260925 22:38:45.251829 21947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:38:45.272387 21947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:38:45.276044 21947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:38:45.276398 21947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:38:45.284922 21947 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.254:43017
I20260925 22:38:45.285233 22014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.254:43017 every 8 connection(s)
I20260925 22:38:45.286245 21947 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
I20260925 22:38:45.288125 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 21947
I20260925 22:38:45.288663 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.193:33913
--local_ip_for_outbound_sockets=127.6.246.193
--tserver_master_addrs=127.6.246.254:43017
--webserver_port=38055
--webserver_interface=127.6.246.193
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.193.keytab
--principal=kudu/127.6.246.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:34883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:38:45.291669 22015 sys_catalog.cc:263] Verifying existing consensus state
I20260925 22:38:45.293675 22015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 761938f0951e48a0910db98fde88b05f: Bootstrap starting.
I20260925 22:38:45.314664 22015 log.cc:824] T 00000000000000000000000000000000 P 761938f0951e48a0910db98fde88b05f: Log is configured to *not* fsync() on all Append() calls
I20260925 22:38:45.327491 22015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 761938f0951e48a0910db98fde88b05f: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 22:38:45.327973 22015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 761938f0951e48a0910db98fde88b05f: Bootstrap complete.
I20260925 22:38:45.334303 22015 raft_consensus.cc:348] T 00000000000000000000000000000000 P 761938f0951e48a0910db98fde88b05f [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "761938f0951e48a0910db98fde88b05f" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 43017 } }
I20260925 22:38:45.334797 22015 raft_consensus.cc:729] T 00000000000000000000000000000000 P 761938f0951e48a0910db98fde88b05f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 761938f0951e48a0910db98fde88b05f, State: Initialized, Role: FOLLOWER
I20260925 22:38:45.335433 22015 consensus_queue.cc:260] T 00000000000000000000000000000000 P 761938f0951e48a0910db98fde88b05f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "761938f0951e48a0910db98fde88b05f" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 43017 } }
I20260925 22:38:45.335600 22015 raft_consensus.cc:388] T 00000000000000000000000000000000 P 761938f0951e48a0910db98fde88b05f [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:38:45.335691 22015 raft_consensus.cc:482] T 00000000000000000000000000000000 P 761938f0951e48a0910db98fde88b05f [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:38:45.335811 22015 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 761938f0951e48a0910db98fde88b05f [term 2 FOLLOWER]: Advancing to term 3
I20260925 22:38:45.338169 22015 raft_consensus.cc:504] T 00000000000000000000000000000000 P 761938f0951e48a0910db98fde88b05f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "761938f0951e48a0910db98fde88b05f" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 43017 } }
I20260925 22:38:45.338649 22015 leader_election.cc:302] T 00000000000000000000000000000000 P 761938f0951e48a0910db98fde88b05f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 761938f0951e48a0910db98fde88b05f; no voters:
I20260925 22:38:45.339051 22015 leader_election.cc:288] T 00000000000000000000000000000000 P 761938f0951e48a0910db98fde88b05f [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 22:38:45.339170 22019 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 761938f0951e48a0910db98fde88b05f [term 3 FOLLOWER]: Leader election won for term 3
I20260925 22:38:45.339389 22019 raft_consensus.cc:686] T 00000000000000000000000000000000 P 761938f0951e48a0910db98fde88b05f [term 3 LEADER]: Becoming Leader. State: Replica: 761938f0951e48a0910db98fde88b05f, State: Running, Role: LEADER
I20260925 22:38:45.339742 22019 consensus_queue.cc:237] T 00000000000000000000000000000000 P 761938f0951e48a0910db98fde88b05f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "761938f0951e48a0910db98fde88b05f" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 43017 } }
I20260925 22:38:45.340626 22015 sys_catalog.cc:564] T 00000000000000000000000000000000 P 761938f0951e48a0910db98fde88b05f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:38:45.340898 22019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 761938f0951e48a0910db98fde88b05f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 761938f0951e48a0910db98fde88b05f. Latest consensus state: current_term: 3 leader_uuid: "761938f0951e48a0910db98fde88b05f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "761938f0951e48a0910db98fde88b05f" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 43017 } } }
I20260925 22:38:45.341005 22019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 761938f0951e48a0910db98fde88b05f [sys.catalog]: This master's current role is: LEADER
I20260925 22:38:45.341325 22020 sys_catalog.cc:455] T 00000000000000000000000000000000 P 761938f0951e48a0910db98fde88b05f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "761938f0951e48a0910db98fde88b05f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "761938f0951e48a0910db98fde88b05f" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 43017 } } }
I20260925 22:38:45.341428 22020 sys_catalog.cc:458] T 00000000000000000000000000000000 P 761938f0951e48a0910db98fde88b05f [sys.catalog]: This master's current role is: LEADER
I20260925 22:38:45.343185 22023 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:38:45.349634 22023 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:38:45.350446 22023 catalog_manager.cc:1259] Loaded cluster ID: 79b9eef78b7641a9a88d8832afd1e153
I20260925 22:38:45.350522 22023 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260925 22:38:45.351719 22023 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f3b164d89ad google::LogMessage::Fail() at ??:0
@ 0x7f3b164db88a google::LogMessage::SendToLog() at ??:0
@ 0x7f3b164d84de google::LogMessage::Flush() at ??:0
@ 0x7f3b164d9a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f3b20e17fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f3b20e18d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f3b20e15579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f3b20e59640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f3b1872a3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f3b1872ac83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f3b1872c3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f3b1871b512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f3b19d5f6db start_thread at ??:0
@ 0x7f3b14fd661f clone at ??:0
W20260925 22:38:45.466372 22017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:38:45.466859 22017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:38:45.466941 22017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:38:45.473477 22017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:38:45.473595 22017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:38:45.473697 22017 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.193
I20260925 22:38:45.482859 22017 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:34883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.193.keytab
--principal=kudu/127.6.246.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.193:33913
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.246.193
--webserver_port=38055
--tserver_master_addrs=127.6.246.254:43017
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.193
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:38:45.484863 22017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:38:45.486505 22017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:38:45.499990 22056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:38:45.500151 22059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:38:45.500013 22057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:38:45.500488 22017 server_base.cc:1034] running on GCE node
I20260925 22:38:45.501230 22017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:38:45.502111 22017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:38:45.503331 22017 hybrid_clock.cc:648] HybridClock initialized: now 1790375925503306 us; error 51 us; skew 500 ppm
Sep 25 22:38:45 dist-test-slave-wjdr krb5kdc[20870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375925, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:38:45.506767 22017 init.cc:377] Logged in from keytab as kudu/127.6.246.193@KRBTEST.COM (short username kudu)
I20260925 22:38:45.507949 22017 webserver.cc:460] Webserver started at http://127.6.246.193:38055/ using document root <none> and password file <none>
I20260925 22:38:45.508757 22017 fs_manager.cc:359] Metadata directory not provided
I20260925 22:38:45.508853 22017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:38:45.514384 22017 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 22:38:45.517153 22066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:38:45.518833 22017 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:38:45.518993 22017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-0/wal
uuid: "593624e63fab46768718c022af35acde"
format_stamp: "Formatted at 2026-09-25 22:38:42 on dist-test-slave-wjdr"
server_key: "65166019af2a3076ec1047c39d91369d"
server_key_iv: "abccb242a4c3c976be25e55b993397a4"
server_key_version: "encryptionkey@0"
I20260925 22:38:45.519711 22017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:38:45.536886 22017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:38:45.541131 22017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:38:45.541394 22017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:38:45.542183 22017 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:38:45.543596 22073 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 22:38:45.549994 22017 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 22:38:45.550146 22017 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20260925 22:38:45.550256 22017 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 22:38:45.552562 22017 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 22:38:45.552640 22017 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.003s
I20260925 22:38:45.552966 22073 tablet_bootstrap.cc:492] T a209003a4bdb4c28b446d2168ead92f2 P 593624e63fab46768718c022af35acde: Bootstrap starting.
I20260925 22:38:45.567317 22017 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.193:33913
I20260925 22:38:45.567588 22180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.193:33913 every 8 connection(s)
I20260925 22:38:45.569206 22017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
I20260925 22:38:45.578377 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 22017
I20260925 22:38:45.578758 22073 log.cc:824] T a209003a4bdb4c28b446d2168ead92f2 P 593624e63fab46768718c022af35acde: Log is configured to *not* fsync() on all Append() calls
I20260925 22:38:45.578963 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.194:45893
--local_ip_for_outbound_sockets=127.6.246.194
--tserver_master_addrs=127.6.246.254:43017
--webserver_port=36019
--webserver_interface=127.6.246.194
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.194.keytab
--principal=kudu/127.6.246.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:34883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:38:45.588544 22073 tablet_bootstrap.cc:492] T a209003a4bdb4c28b446d2168ead92f2 P 593624e63fab46768718c022af35acde: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 22:38:45.589123 22073 tablet_bootstrap.cc:492] T a209003a4bdb4c28b446d2168ead92f2 P 593624e63fab46768718c022af35acde: Bootstrap complete.
I20260925 22:38:45.590683 22073 ts_tablet_manager.cc:1397] T a209003a4bdb4c28b446d2168ead92f2 P 593624e63fab46768718c022af35acde: Time spent bootstrapping tablet: real 0.038s user 0.026s sys 0.009s
I20260925 22:38:45.596316 22073 raft_consensus.cc:348] T a209003a4bdb4c28b446d2168ead92f2 P 593624e63fab46768718c022af35acde [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f130a10f1e64d5e8439fa19988dfee5" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43353 } } peers { permanent_uuid: "6487020f6fdd432eaecee9f7c1c8beb1" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 45893 } } peers { permanent_uuid: "593624e63fab46768718c022af35acde" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 33913 } }
I20260925 22:38:45.597255 22073 raft_consensus.cc:729] T a209003a4bdb4c28b446d2168ead92f2 P 593624e63fab46768718c022af35acde [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 593624e63fab46768718c022af35acde, State: Initialized, Role: FOLLOWER
I20260925 22:38:45.598089 22073 consensus_queue.cc:260] T a209003a4bdb4c28b446d2168ead92f2 P 593624e63fab46768718c022af35acde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f130a10f1e64d5e8439fa19988dfee5" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43353 } } peers { permanent_uuid: "6487020f6fdd432eaecee9f7c1c8beb1" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 45893 } } peers { permanent_uuid: "593624e63fab46768718c022af35acde" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 33913 } }
I20260925 22:38:45.599761 22073 ts_tablet_manager.cc:1428] T a209003a4bdb4c28b446d2168ead92f2 P 593624e63fab46768718c022af35acde: Time spent starting tablet: real 0.009s user 0.010s sys 0.000s
W20260925 22:38:45.614027 22183 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:38:45.572333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43017 (local address 127.6.246.193:51127)
0925 22:38:45.573114 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:45.573188 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0925 22:38:45.574556 (+ 1368us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:38:45.612901 (+ 38345us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43017: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":203,"threads_started":1}
W20260925 22:38:45.614717 22181 heartbeater.cc:644] Failed to heartbeat to 127.6.246.254:43017 (0 consecutive failures): Network error: Failed to ping master at 127.6.246.254:43017: Client connection negotiation failed: client connection to 127.6.246.254:43017: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 22:38:45.615510 22183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:45.615233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43017 (local address 127.6.246.193:51789)
0925 22:38:45.615372 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:45.615456 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260925 22:38:45.616147 22183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:45.615937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43017 (local address 127.6.246.193:41817)
0925 22:38:45.616044 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:45.616092 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260925 22:38:45.616304 22181 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 22:38:45.723182 22185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:38:45.723583 22185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:38:45.723690 22185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:38:45.728052 22185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:38:45.728160 22185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:38:45.728261 22185 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.194
I20260925 22:38:45.734158 22185 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:34883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.194.keytab
--principal=kudu/127.6.246.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.194:45893
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.246.194
--webserver_port=36019
--tserver_master_addrs=127.6.246.254:43017
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22185
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.194
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:38:45.735615 22185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:38:45.736935 22185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:38:45.748628 22192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:38:45.748628 22191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:38:45.748649 22194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:38:45.748994 22185 server_base.cc:1034] running on GCE node
I20260925 22:38:45.749473 22185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:38:45.750231 22185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:38:45.751411 22185 hybrid_clock.cc:648] HybridClock initialized: now 1790375925751374 us; error 48 us; skew 500 ppm
Sep 25 22:38:45 dist-test-slave-wjdr krb5kdc[20870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375925, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:38:45.754731 22185 init.cc:377] Logged in from keytab as kudu/127.6.246.194@KRBTEST.COM (short username kudu)
I20260925 22:38:45.755795 22185 webserver.cc:460] Webserver started at http://127.6.246.194:36019/ using document root <none> and password file <none>
I20260925 22:38:45.756544 22185 fs_manager.cc:359] Metadata directory not provided
I20260925 22:38:45.756614 22185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:38:45.761222 22185 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:38:45.763785 22201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:38:45.765398 22185 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:38:45.765513 22185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-1/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-1/wal
uuid: "6487020f6fdd432eaecee9f7c1c8beb1"
format_stamp: "Formatted at 2026-09-25 22:38:43 on dist-test-slave-wjdr"
server_key: "4642dbdfdadc8575fad1e4e9b7d083e8"
server_key_iv: "c5ee12b563ba3927f8ba289716028424"
server_key_version: "encryptionkey@0"
I20260925 22:38:45.766199 22185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:38:45.796056 22185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:38:45.800148 22185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:38:45.800401 22185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:38:45.801254 22185 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:38:45.802552 22185 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:38:45.802615 22185 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:38:45.802700 22185 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:38:45.802742 22185 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:38:45.813104 22185 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.194:45893
I20260925 22:38:45.813130 22314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.194:45893 every 8 connection(s)
I20260925 22:38:45.814391 22185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
I20260925 22:38:45.815593 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 22185
I20260925 22:38:45.816085 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.195:43353
--local_ip_for_outbound_sockets=127.6.246.195
--tserver_master_addrs=127.6.246.254:43017
--webserver_port=45167
--webserver_interface=127.6.246.195
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.195.keytab
--principal=kudu/127.6.246.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:34883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:38:45.819231 22318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:45.817248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43017 (local address 127.6.246.194:41841)
0925 22:38:45.818210 (+ 962us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:45.818340 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":698,"thread_start_us":172,"threads_started":1}
W20260925 22:38:45.819753 22315 heartbeater.cc:644] Failed to heartbeat to 127.6.246.254:43017 (0 consecutive failures): Network error: Failed to ping master at 127.6.246.254:43017: Client connection negotiation failed: client connection to 127.6.246.254:43017: connect: Connection refused (error 111)
I20260925 22:38:45.820675 22318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:45.820270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43017 (local address 127.6.246.194:38835)
0925 22:38:45.820434 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:45.820553 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260925 22:38:45.821514 22318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:45.821284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43017 (local address 127.6.246.194:41083)
0925 22:38:45.821400 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:45.821457 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 22:38:45.821767 22315 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 22:38:45.946590 22317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:38:45.946967 22317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:38:45.947019 22317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:38:45.951486 22317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:38:45.951594 22317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:38:45.951666 22317 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.195
I20260925 22:38:45.957758 22317 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:34883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.195.keytab
--principal=kudu/127.6.246.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.195:43353
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.246.195
--webserver_port=45167
--tserver_master_addrs=127.6.246.254:43017
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22317
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.195
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:38:45.959193 22317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:38:45.960537 22317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:38:45.972096 22323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:38:45.972177 22324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:38:45.972096 22326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:38:45.972589 22317 server_base.cc:1034] running on GCE node
I20260925 22:38:45.973127 22317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:38:45.973918 22317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:38:45.975139 22317 hybrid_clock.cc:648] HybridClock initialized: now 1790375925975119 us; error 33 us; skew 500 ppm
Sep 25 22:38:45 dist-test-slave-wjdr krb5kdc[20870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375925, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:38:45.978322 22317 init.cc:377] Logged in from keytab as kudu/127.6.246.195@KRBTEST.COM (short username kudu)
I20260925 22:38:45.979380 22317 webserver.cc:460] Webserver started at http://127.6.246.195:45167/ using document root <none> and password file <none>
I20260925 22:38:45.980079 22317 fs_manager.cc:359] Metadata directory not provided
I20260925 22:38:45.980175 22317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:38:45.984946 22317 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:38:45.987659 22333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:38:45.989198 22317 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:38:45.989342 22317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-2/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-2/wal
uuid: "2f130a10f1e64d5e8439fa19988dfee5"
format_stamp: "Formatted at 2026-09-25 22:38:43 on dist-test-slave-wjdr"
server_key: "07c03525d4e9c4e6ce3969574621eb3c"
server_key_iv: "2235cc6693edd0c629bd98ee65b2c830"
server_key_version: "encryptionkey@0"
I20260925 22:38:45.990008 22317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:38:46.006629 22317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:38:46.010802 22317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:38:46.011051 22317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:38:46.011811 22317 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:38:46.012895 22317 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:38:46.012948 22317 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:38:46.013016 22317 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:38:46.013056 22317 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:38:46.023303 22317 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.195:43353
I20260925 22:38:46.023351 22446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.195:43353 every 8 connection(s)
I20260925 22:38:46.024777 22317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
I20260925 22:38:46.025153 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 22317
I20260925 22:38:46.029258 22450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:46.027431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43017 (local address 127.6.246.195:33421)
0925 22:38:46.028224 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:46.028338 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":135,"threads_started":1}
W20260925 22:38:46.029772 22447 heartbeater.cc:644] Failed to heartbeat to 127.6.246.254:43017 (0 consecutive failures): Network error: Failed to ping master at 127.6.246.254:43017: Client connection negotiation failed: client connection to 127.6.246.254:43017: connect: Connection refused (error 111)
I20260925 22:38:46.030350 22450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:46.030149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43017 (local address 127.6.246.195:51411)
0925 22:38:46.030252 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:46.030294 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 22:38:46.030973 22450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:46.030792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43017 (local address 127.6.246.195:60821)
0925 22:38:46.030882 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:46.030925 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260925 22:38:46.031217 22447 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 22:38:46.617870 22451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:46.617216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43017 (local address 127.6.246.193:55171)
0925 22:38:46.617643 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:46.617738 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":109,"threads_started":1}
I20260925 22:38:46.823122 22452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:46.822558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43017 (local address 127.6.246.194:39485)
0925 22:38:46.822879 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:46.822957 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":82,"threads_started":1}
I20260925 22:38:46.964169 22453 raft_consensus.cc:482] T a209003a4bdb4c28b446d2168ead92f2 P 593624e63fab46768718c022af35acde [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:38:46.964564 22453 raft_consensus.cc:504] T a209003a4bdb4c28b446d2168ead92f2 P 593624e63fab46768718c022af35acde [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f130a10f1e64d5e8439fa19988dfee5" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43353 } } peers { permanent_uuid: "6487020f6fdd432eaecee9f7c1c8beb1" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 45893 } } peers { permanent_uuid: "593624e63fab46768718c022af35acde" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 33913 } }
I20260925 22:38:46.966230 22453 leader_election.cc:288] T a209003a4bdb4c28b446d2168ead92f2 P 593624e63fab46768718c022af35acde [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2f130a10f1e64d5e8439fa19988dfee5 (127.6.246.195:43353), 6487020f6fdd432eaecee9f7c1c8beb1 (127.6.246.194:45893)
Sep 25 22:38:46 dist-test-slave-wjdr krb5kdc[20870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375925, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.193@KRBTEST.COM for kudu/127.6.246.195@KRBTEST.COM
Sep 25 22:38:46 dist-test-slave-wjdr krb5kdc[20870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375925, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.193@KRBTEST.COM for kudu/127.6.246.194@KRBTEST.COM
I20260925 22:38:46.984146 22451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:46.966502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.194:45893 (local address 127.6.246.193:59465)
0925 22:38:46.966848 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:46.966872 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 22:38:46.966992 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:38:46.969165 (+ 2173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:38:46.969173 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:38:46.969928 (+ 755us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:38:46.970513 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:38:46.970529 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:38:46.972021 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:38:46.972025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:38:46.973227 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:38:46.973238 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:38:46.974022 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:38:46.974026 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:38:46.974351 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:46.975125 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:38:46.975143 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:38:46.977458 (+ 2315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:38:46.980490 (+ 3032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:38:46.980499 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:38:46.980526 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:38:46.981106 (+ 580us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:38:46.981542 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:38:46.981545 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:38:46.981548 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:38:46.981781 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:38:46.982673 (+ 892us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:38:46.982681 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:38:46.983154 (+ 473us) client_negotiation.cc:769] Sending connection context
0925 22:38:46.983407 (+ 253us) client_negotiation.cc:240] Negotiation successful
0925 22:38:46.983932 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":378,"spinlock_wait_cycles":34816}
I20260925 22:38:46.984207 22456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:46.967006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.193:59823 (local address 127.6.246.195:43353)
0925 22:38:46.967386 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0925 22:38:46.967403 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 22:38:46.967446 (+ 43us) server_negotiation.cc:407] Connection header received
0925 22:38:46.968249 (+ 803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:38:46.968255 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:38:46.968741 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:38:46.969119 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:38:46.970704 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:46.971884 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:46.973351 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:46.973875 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:46.974085 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:46.977732 (+ 3647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:38:46.977739 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:38:46.977749 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:38:46.977792 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:38:46.980300 (+ 2508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:38:46.981254 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:38:46.981261 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:38:46.981270 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:38:46.981428 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:38:46.981905 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:38:46.981908 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:38:46.981910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:38:46.982513 (+ 603us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:38:46.982662 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 22:38:46.983606 (+ 944us) server_negotiation.cc:299] Negotiation successful
0925 22:38:46.983979 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":111,"threads_started":1}
I20260925 22:38:46.984218 22455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:46.966805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.193:59465 (local address 127.6.246.194:45893)
0925 22:38:46.967357 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0925 22:38:46.967391 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0925 22:38:46.967438 (+ 47us) server_negotiation.cc:407] Connection header received
0925 22:38:46.968222 (+ 784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:38:46.968229 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:38:46.968740 (+ 511us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:38:46.969119 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:38:46.970700 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:46.971877 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:46.973371 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:38:46.973891 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:38:46.974077 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:46.977663 (+ 3586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:38:46.977672 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:38:46.977689 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:38:46.977725 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:38:46.980282 (+ 2557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:38:46.981254 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:38:46.981261 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:38:46.981269 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:38:46.981428 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:38:46.981905 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:38:46.981909 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:38:46.981911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:38:46.982513 (+ 602us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:38:46.982662 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 22:38:46.983606 (+ 944us) server_negotiation.cc:299] Negotiation successful
0925 22:38:46.983979 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":111,"threads_started":1}
I20260925 22:38:46.984146 22454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:46.966502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.195:43353 (local address 127.6.246.193:59823)
0925 22:38:46.966981 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:46.966997 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:38:46.967152 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:38:46.969165 (+ 2013us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:38:46.969172 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:38:46.969928 (+ 756us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:38:46.970513 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:38:46.970530 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:38:46.972025 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:38:46.972031 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:38:46.973222 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:38:46.973231 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:38:46.974024 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:38:46.974027 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:38:46.974351 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:38:46.975178 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:38:46.975191 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:38:46.977105 (+ 1914us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:38:46.980490 (+ 3385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:38:46.980499 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:38:46.980520 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:38:46.981106 (+ 586us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:38:46.981542 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:38:46.981545 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:38:46.981548 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:38:46.981781 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:38:46.982673 (+ 892us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:38:46.982681 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:38:46.983154 (+ 473us) client_negotiation.cc:769] Sending connection context
0925 22:38:46.983407 (+ 253us) client_negotiation.cc:240] Negotiation successful
0925 22:38:46.983875 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"mutex_wait_us":390,"spinlock_wait_cycles":68992,"thread_start_us":71,"threads_started":1}
W20260925 22:38:46.987164 22069 leader_election.cc:341] T a209003a4bdb4c28b446d2168ead92f2 P 593624e63fab46768718c022af35acde [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6487020f6fdd432eaecee9f7c1c8beb1 (127.6.246.194:45893): Not found: Tablet not found: a209003a4bdb4c28b446d2168ead92f2
W20260925 22:38:46.987363 22068 leader_election.cc:341] T a209003a4bdb4c28b446d2168ead92f2 P 593624e63fab46768718c022af35acde [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2f130a10f1e64d5e8439fa19988dfee5 (127.6.246.195:43353): Not found: Tablet not found: a209003a4bdb4c28b446d2168ead92f2
I20260925 22:38:46.987430 22069 leader_election.cc:302] T a209003a4bdb4c28b446d2168ead92f2 P 593624e63fab46768718c022af35acde [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 593624e63fab46768718c022af35acde; no voters: 2f130a10f1e64d5e8439fa19988dfee5, 6487020f6fdd432eaecee9f7c1c8beb1
I20260925 22:38:46.987804 22453 raft_consensus.cc:2726] T a209003a4bdb4c28b446d2168ead92f2 P 593624e63fab46768718c022af35acde [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 22:38:47.032604 22457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:47.031931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43017 (local address 127.6.246.195:36209)
0925 22:38:47.032286 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:47.032406 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":92,"threads_started":1}
I20260925 22:38:47.619434 22458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:47.618854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43017 (local address 127.6.246.193:51503)
0925 22:38:47.619216 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:47.619296 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":101,"threads_started":1}
I20260925 22:38:47.824908 22459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:47.824242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43017 (local address 127.6.246.194:36007)
0925 22:38:47.824672 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:47.824762 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":111,"threads_started":1}
I20260925 22:38:48.033901 22460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:48.033397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43017 (local address 127.6.246.195:36433)
0925 22:38:48.033695 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:48.033774 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":92,"threads_started":1}
I20260925 22:38:48.621120 22461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:48.620541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43017 (local address 127.6.246.193:34869)
0925 22:38:48.620912 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:48.620996 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":99,"threads_started":1}
I20260925 22:38:48.660837 22462 raft_consensus.cc:482] T a209003a4bdb4c28b446d2168ead92f2 P 593624e63fab46768718c022af35acde [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:38:48.660991 22462 raft_consensus.cc:504] T a209003a4bdb4c28b446d2168ead92f2 P 593624e63fab46768718c022af35acde [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f130a10f1e64d5e8439fa19988dfee5" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43353 } } peers { permanent_uuid: "6487020f6fdd432eaecee9f7c1c8beb1" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 45893 } } peers { permanent_uuid: "593624e63fab46768718c022af35acde" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 33913 } }
I20260925 22:38:48.661530 22462 leader_election.cc:288] T a209003a4bdb4c28b446d2168ead92f2 P 593624e63fab46768718c022af35acde [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2f130a10f1e64d5e8439fa19988dfee5 (127.6.246.195:43353), 6487020f6fdd432eaecee9f7c1c8beb1 (127.6.246.194:45893)
W20260925 22:38:48.662500 22069 leader_election.cc:341] T a209003a4bdb4c28b446d2168ead92f2 P 593624e63fab46768718c022af35acde [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6487020f6fdd432eaecee9f7c1c8beb1 (127.6.246.194:45893): Not found: Tablet not found: a209003a4bdb4c28b446d2168ead92f2
W20260925 22:38:48.662628 22068 leader_election.cc:341] T a209003a4bdb4c28b446d2168ead92f2 P 593624e63fab46768718c022af35acde [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2f130a10f1e64d5e8439fa19988dfee5 (127.6.246.195:43353): Not found: Tablet not found: a209003a4bdb4c28b446d2168ead92f2
I20260925 22:38:48.662765 22068 leader_election.cc:302] T a209003a4bdb4c28b446d2168ead92f2 P 593624e63fab46768718c022af35acde [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 593624e63fab46768718c022af35acde; no voters: 2f130a10f1e64d5e8439fa19988dfee5, 6487020f6fdd432eaecee9f7c1c8beb1
I20260925 22:38:48.662943 22462 raft_consensus.cc:2726] T a209003a4bdb4c28b446d2168ead92f2 P 593624e63fab46768718c022af35acde [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 22:38:48.826637 22463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:48.825949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43017 (local address 127.6.246.194:41963)
0925 22:38:48.826359 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:48.826460 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":118,"threads_started":1}
I20260925 22:38:49.035485 22464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:49.034887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43017 (local address 127.6.246.195:46723)
0925 22:38:49.035307 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:49.035379 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":162,"threads_started":1}
I20260925 22:38:49.622668 22465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:49.622090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43017 (local address 127.6.246.193:56451)
0925 22:38:49.622457 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:49.622539 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":166,"threads_started":1}
I20260925 22:38:49.828048 22466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:49.827572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43017 (local address 127.6.246.194:39793)
0925 22:38:49.827884 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:49.827944 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":106,"threads_started":1}
I20260925 22:38:50.037110 22467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:50.036558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43017 (local address 127.6.246.195:46839)
0925 22:38:50.036887 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:50.036976 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":83,"threads_started":1}
I20260925 22:38:50.170360 22468 raft_consensus.cc:482] T a209003a4bdb4c28b446d2168ead92f2 P 593624e63fab46768718c022af35acde [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:38:50.170570 22468 raft_consensus.cc:504] T a209003a4bdb4c28b446d2168ead92f2 P 593624e63fab46768718c022af35acde [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f130a10f1e64d5e8439fa19988dfee5" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43353 } } peers { permanent_uuid: "6487020f6fdd432eaecee9f7c1c8beb1" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 45893 } } peers { permanent_uuid: "593624e63fab46768718c022af35acde" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 33913 } }
I20260925 22:38:50.171162 22468 leader_election.cc:288] T a209003a4bdb4c28b446d2168ead92f2 P 593624e63fab46768718c022af35acde [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2f130a10f1e64d5e8439fa19988dfee5 (127.6.246.195:43353), 6487020f6fdd432eaecee9f7c1c8beb1 (127.6.246.194:45893)
W20260925 22:38:50.172087 22068 leader_election.cc:341] T a209003a4bdb4c28b446d2168ead92f2 P 593624e63fab46768718c022af35acde [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2f130a10f1e64d5e8439fa19988dfee5 (127.6.246.195:43353): Not found: Tablet not found: a209003a4bdb4c28b446d2168ead92f2
W20260925 22:38:50.172207 22069 leader_election.cc:341] T a209003a4bdb4c28b446d2168ead92f2 P 593624e63fab46768718c022af35acde [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6487020f6fdd432eaecee9f7c1c8beb1 (127.6.246.194:45893): Not found: Tablet not found: a209003a4bdb4c28b446d2168ead92f2
I20260925 22:38:50.172370 22069 leader_election.cc:302] T a209003a4bdb4c28b446d2168ead92f2 P 593624e63fab46768718c022af35acde [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 593624e63fab46768718c022af35acde; no voters: 2f130a10f1e64d5e8439fa19988dfee5, 6487020f6fdd432eaecee9f7c1c8beb1
I20260925 22:38:50.172569 22468 raft_consensus.cc:2726] T a209003a4bdb4c28b446d2168ead92f2 P 593624e63fab46768718c022af35acde [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 22:38:50.624496 22469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:50.623873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43017 (local address 127.6.246.193:43719)
0925 22:38:50.624257 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:50.624323 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":124,"threads_started":1}
I20260925 22:38:50.829612 22470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:50.829044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43017 (local address 127.6.246.194:56255)
0925 22:38:50.829384 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:50.829451 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":127,"threads_started":1}
I20260925 22:38:51.038650 22471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:51.038135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43017 (local address 127.6.246.195:46981)
0925 22:38:51.038449 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:51.038522 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":106,"threads_started":1}
I20260925 22:38:51.626113 22472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:51.625504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43017 (local address 127.6.246.193:43761)
0925 22:38:51.625894 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:51.625990 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":128,"threads_started":1}
I20260925 22:38:51.831207 22473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:51.830634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43017 (local address 127.6.246.194:54587)
0925 22:38:51.831036 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:51.831098 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":123,"threads_started":1}
I20260925 22:38:52.040343 22474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:52.039763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43017 (local address 127.6.246.195:38969)
0925 22:38:52.040118 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:52.040207 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":101,"threads_started":1}
I20260925 22:38:52.627562 22475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:52.627057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43017 (local address 127.6.246.193:45043)
0925 22:38:52.627361 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:52.627435 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":87,"threads_started":1}
I20260925 22:38:52.832998 22476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:52.832276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43017 (local address 127.6.246.194:42655)
0925 22:38:52.832757 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:52.832842 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":132,"threads_started":1}
I20260925 22:38:53.042150 22477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:53.041475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43017 (local address 127.6.246.195:46695)
0925 22:38:53.041891 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:53.041979 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":107,"threads_started":1}
I20260925 22:38:53.629200 22478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:53.628698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43017 (local address 127.6.246.193:53949)
0925 22:38:53.629009 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:53.629078 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":128,"threads_started":1}
I20260925 22:38:53.834683 22479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:53.833999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43017 (local address 127.6.246.194:59891)
0925 22:38:53.834383 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:53.834495 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":108,"threads_started":1}
I20260925 22:38:54.043879 22480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:54.043204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43017 (local address 127.6.246.195:54871)
0925 22:38:54.043558 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:54.043695 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":98,"threads_started":1}
I20260925 22:38:54.353530 22481 raft_consensus.cc:482] T a209003a4bdb4c28b446d2168ead92f2 P 593624e63fab46768718c022af35acde [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:38:54.353796 22481 raft_consensus.cc:504] T a209003a4bdb4c28b446d2168ead92f2 P 593624e63fab46768718c022af35acde [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f130a10f1e64d5e8439fa19988dfee5" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43353 } } peers { permanent_uuid: "6487020f6fdd432eaecee9f7c1c8beb1" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 45893 } } peers { permanent_uuid: "593624e63fab46768718c022af35acde" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 33913 } }
I20260925 22:38:54.354350 22481 leader_election.cc:288] T a209003a4bdb4c28b446d2168ead92f2 P 593624e63fab46768718c022af35acde [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2f130a10f1e64d5e8439fa19988dfee5 (127.6.246.195:43353), 6487020f6fdd432eaecee9f7c1c8beb1 (127.6.246.194:45893)
W20260925 22:38:54.355491 22069 leader_election.cc:341] T a209003a4bdb4c28b446d2168ead92f2 P 593624e63fab46768718c022af35acde [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6487020f6fdd432eaecee9f7c1c8beb1 (127.6.246.194:45893): Not found: Tablet not found: a209003a4bdb4c28b446d2168ead92f2
W20260925 22:38:54.355669 22068 leader_election.cc:341] T a209003a4bdb4c28b446d2168ead92f2 P 593624e63fab46768718c022af35acde [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2f130a10f1e64d5e8439fa19988dfee5 (127.6.246.195:43353): Not found: Tablet not found: a209003a4bdb4c28b446d2168ead92f2
I20260925 22:38:54.355744 22068 leader_election.cc:302] T a209003a4bdb4c28b446d2168ead92f2 P 593624e63fab46768718c022af35acde [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 593624e63fab46768718c022af35acde; no voters: 2f130a10f1e64d5e8439fa19988dfee5, 6487020f6fdd432eaecee9f7c1c8beb1
I20260925 22:38:54.355906 22481 raft_consensus.cc:2726] T a209003a4bdb4c28b446d2168ead92f2 P 593624e63fab46768718c022af35acde [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 22:38:54.630527 22482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:54.629941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43017 (local address 127.6.246.193:43105)
0925 22:38:54.630305 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:54.630394 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":92,"threads_started":1}
I20260925 22:38:54.836696 22483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:54.835903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43017 (local address 127.6.246.194:38393)
0925 22:38:54.836378 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:54.836530 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":111,"threads_started":1}
I20260925 22:38:55.045785 22484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:55.045183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43017 (local address 127.6.246.195:54905)
0925 22:38:55.045574 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:55.045663 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":103,"threads_started":1}
I20260925 22:38:55.631992 22485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:55.631550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43017 (local address 127.6.246.193:55025)
0925 22:38:55.631828 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:55.631885 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":117,"threads_started":1}
I20260925 22:38:55.838496 22486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:55.837872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43017 (local address 127.6.246.194:51513)
0925 22:38:55.838256 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:55.838342 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":94,"threads_started":1}
I20260925 22:38:56.047585 22487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:56.046898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43017 (local address 127.6.246.195:40031)
0925 22:38:56.047307 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:56.047403 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":137,"threads_started":1}
I20260925 22:38:56.190021 22488 raft_consensus.cc:482] T a209003a4bdb4c28b446d2168ead92f2 P 593624e63fab46768718c022af35acde [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:38:56.190198 22488 raft_consensus.cc:504] T a209003a4bdb4c28b446d2168ead92f2 P 593624e63fab46768718c022af35acde [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f130a10f1e64d5e8439fa19988dfee5" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 43353 } } peers { permanent_uuid: "6487020f6fdd432eaecee9f7c1c8beb1" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 45893 } } peers { permanent_uuid: "593624e63fab46768718c022af35acde" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 33913 } }
I20260925 22:38:56.190783 22488 leader_election.cc:288] T a209003a4bdb4c28b446d2168ead92f2 P 593624e63fab46768718c022af35acde [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2f130a10f1e64d5e8439fa19988dfee5 (127.6.246.195:43353), 6487020f6fdd432eaecee9f7c1c8beb1 (127.6.246.194:45893)
W20260925 22:38:56.191726 22068 leader_election.cc:341] T a209003a4bdb4c28b446d2168ead92f2 P 593624e63fab46768718c022af35acde [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2f130a10f1e64d5e8439fa19988dfee5 (127.6.246.195:43353): Not found: Tablet not found: a209003a4bdb4c28b446d2168ead92f2
W20260925 22:38:56.191829 22069 leader_election.cc:341] T a209003a4bdb4c28b446d2168ead92f2 P 593624e63fab46768718c022af35acde [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6487020f6fdd432eaecee9f7c1c8beb1 (127.6.246.194:45893): Not found: Tablet not found: a209003a4bdb4c28b446d2168ead92f2
I20260925 22:38:56.191893 22069 leader_election.cc:302] T a209003a4bdb4c28b446d2168ead92f2 P 593624e63fab46768718c022af35acde [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 593624e63fab46768718c022af35acde; no voters: 2f130a10f1e64d5e8439fa19988dfee5, 6487020f6fdd432eaecee9f7c1c8beb1
I20260925 22:38:56.192034 22488 raft_consensus.cc:2726] T a209003a4bdb4c28b446d2168ead92f2 P 593624e63fab46768718c022af35acde [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 22:38:56.633644 22489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:56.632986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43017 (local address 127.6.246.193:49153)
0925 22:38:56.633387 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:56.633476 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":103,"threads_started":1}
I20260925 22:38:56.840152 22490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:56.839522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43017 (local address 127.6.246.194:58393)
0925 22:38:56.839912 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:56.839988 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":115,"threads_started":1}
I20260925 22:38:57.049204 22491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:57.048651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43017 (local address 127.6.246.195:36217)
0925 22:38:57.049003 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:57.049076 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":110,"threads_started":1}
I20260925 22:38:57.635140 22492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:57.634595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43017 (local address 127.6.246.193:40567)
0925 22:38:57.634936 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:57.635006 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":98,"threads_started":1}
I20260925 22:38:57.841697 22493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:57.841150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43017 (local address 127.6.246.194:34577)
0925 22:38:57.841471 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:57.841533 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":105,"threads_started":1}
I20260925 22:38:58.050932 22494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:58.050401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43017 (local address 127.6.246.195:46403)
0925 22:38:58.050685 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:58.050749 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":93,"threads_started":1}
I20260925 22:38:58.636759 22495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:58.636118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43017 (local address 127.6.246.193:49641)
0925 22:38:58.636557 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:58.636630 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":124,"threads_started":1}
I20260925 22:38:58.843351 22496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:58.842785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43017 (local address 127.6.246.194:45569)
0925 22:38:58.843140 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:58.843219 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":102,"threads_started":1}
I20260925 22:38:59.052628 22497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:59.051980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43017 (local address 127.6.246.195:43305)
0925 22:38:59.052335 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:59.052413 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":99,"threads_started":1}
I20260925 22:38:59.638166 22498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:59.637646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43017 (local address 127.6.246.193:58185)
0925 22:38:59.637975 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:59.638039 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":120,"threads_started":1}
I20260925 22:38:59.845147 22499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:38:59.844577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43017 (local address 127.6.246.194:35359)
0925 22:38:59.844924 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0925 22:38:59.845010 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":93,"threads_started":1}
I20260925 22:39:00.054383 22500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:00.053692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43017 (local address 127.6.246.195:53011)
0925 22:39:00.054105 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0925 22:39:00.054209 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":131,"threads_started":1}
I20260925 22:39:00.640013 22501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:00.639422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43017 (local address 127.6.246.193:38215)
0925 22:39:00.639786 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0925 22:39:00.639898 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":107,"threads_started":1}
I20260925 22:39:00.846654 22502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:00.846182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43017 (local address 127.6.246.194:59333)
0925 22:39:00.846472 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0925 22:39:00.846531 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":95,"threads_started":1}
I20260925 22:39:01.032371 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 22017
I20260925 22:39:01.039930 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 22185
I20260925 22:39:01.046682 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 22317
2026-09-25T22:39:01Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19974 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:39:01 dist-test-slave-wjdr krb5kdc[22505](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:39:01 dist-test-slave-wjdr krb5kdc[22505](info): set up 2 sockets
Sep 25 22:39:01 dist-test-slave-wjdr krb5kdc[22505](info): commencing operation
krb5kdc: starting...
W20260925 22:39:02.105417 7131 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Sep 25 22:39:02 dist-test-slave-wjdr krb5kdc[22505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:39:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:39:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.246.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab.
Entry for principal kudu/127.6.246.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab.
I20260925 22:39:02.240684 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:43767
--webserver_interface=127.6.246.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab
--principal=kudu/127.6.246.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:33735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.246.254:43767
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:39:02.371812 22518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:39:02.372174 22518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:39:02.372267 22518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:39:02.376382 22518 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 22:39:02.376469 22518 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 22:39:02.376518 22518 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 22:39:02.376551 22518 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:39:02.376595 22518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:39:02.376632 22518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:39:02.376672 22518 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:39:02.376701 22518 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:39:02.382270 22518 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:33735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.6.246.254:43767
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab
--principal=kudu/127.6.246.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:43767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.246.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:39:02.383651 22518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:39:02.384997 22518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:39:02.395761 22524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:39:02.395774 22523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:39:02.395834 22518 server_base.cc:1034] running on GCE node
W20260925 22:39:02.395857 22526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:39:02.396936 22518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:39:02.398391 22518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:39:02.399643 22518 hybrid_clock.cc:648] HybridClock initialized: now 1790375942399610 us; error 45 us; skew 500 ppm
Sep 25 22:39:02 dist-test-slave-wjdr krb5kdc[22505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375942, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:39:02.402983 22518 init.cc:377] Logged in from keytab as kudu/127.6.246.254@KRBTEST.COM (short username kudu)
I20260925 22:39:02.404069 22518 webserver.cc:460] Webserver started at http://127.6.246.254:39397/ using document root <none> and password file <none>
I20260925 22:39:02.404825 22518 fs_manager.cc:359] Metadata directory not provided
I20260925 22:39:02.404892 22518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:39:02.405102 22518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:39:02.408382 22518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/master-0/data/instance:
uuid: "c836969096ff45db811a88fa616fd000"
format_stamp: "Formatted at 2026-09-25 22:39:02 on dist-test-slave-wjdr"
server_key: "52344f895b6f9d8f0459de73d0663882"
server_key_iv: "970d6b85cc71460be704c5c6758a4c31"
server_key_version: "encryptionkey@0"
I20260925 22:39:02.409242 22518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/master-0/wal/instance:
uuid: "c836969096ff45db811a88fa616fd000"
format_stamp: "Formatted at 2026-09-25 22:39:02 on dist-test-slave-wjdr"
server_key: "52344f895b6f9d8f0459de73d0663882"
server_key_iv: "970d6b85cc71460be704c5c6758a4c31"
server_key_version: "encryptionkey@0"
I20260925 22:39:02.414004 22518 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 22:39:02.416960 22533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:39:02.418476 22518 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:39:02.418630 22518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/master-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/master-0/wal
uuid: "c836969096ff45db811a88fa616fd000"
format_stamp: "Formatted at 2026-09-25 22:39:02 on dist-test-slave-wjdr"
server_key: "52344f895b6f9d8f0459de73d0663882"
server_key_iv: "970d6b85cc71460be704c5c6758a4c31"
server_key_version: "encryptionkey@0"
I20260925 22:39:02.418757 22518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:39:02.438822 22518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:39:02.442430 22518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:39:02.442673 22518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:39:02.451099 22518 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.254:43767
I20260925 22:39:02.451435 22585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.254:43767 every 8 connection(s)
I20260925 22:39:02.452365 22518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
I20260925 22:39:02.456014 22586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:39:02.459682 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 22518
I20260925 22:39:02.459838 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/master-0/wal/instance
I20260925 22:39:02.460242 7131 external_mini_cluster.cc:1428] Setting key 781e65a37145b7a52e73f459fa4c12a8
I20260925 22:39:02.463884 22586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c836969096ff45db811a88fa616fd000: Bootstrap starting.
I20260925 22:39:02.467286 22586 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c836969096ff45db811a88fa616fd000: Neither blocks nor log segments found. Creating new log.
I20260925 22:39:02.468386 22586 log.cc:824] T 00000000000000000000000000000000 P c836969096ff45db811a88fa616fd000: Log is configured to *not* fsync() on all Append() calls
Sep 25 22:39:02 dist-test-slave-wjdr krb5kdc[22505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
I20260925 22:39:02.471163 22586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c836969096ff45db811a88fa616fd000: No bootstrap required, opened a new log
I20260925 22:39:02.476923 22586 raft_consensus.cc:348] T 00000000000000000000000000000000 P c836969096ff45db811a88fa616fd000 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c836969096ff45db811a88fa616fd000" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 43767 } }
I20260925 22:39:02.477252 22586 raft_consensus.cc:374] T 00000000000000000000000000000000 P c836969096ff45db811a88fa616fd000 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:39:02.477299 22589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:02.462049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41488 (local address 127.6.246.254:43767)
0925 22:39:02.462547 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0925 22:39:02.462573 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0925 22:39:02.462618 (+ 45us) server_negotiation.cc:407] Connection header received
0925 22:39:02.463379 (+ 761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:39:02.463398 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:39:02.463878 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:39:02.464277 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:39:02.465288 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:39:02.466366 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:39:02.467554 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:39:02.468175 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:39:02.468309 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:39:02.471470 (+ 3161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:39:02.471478 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:39:02.471498 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:39:02.471530 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:39:02.473708 (+ 2178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:39:02.474195 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:39:02.474203 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:39:02.474211 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:39:02.474394 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:39:02.474665 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:39:02.474668 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:39:02.474672 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:39:02.475263 (+ 591us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:39:02.475504 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0925 22:39:02.475706 (+ 202us) server_negotiation.cc:299] Negotiation successful
0925 22:39:02.476079 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":154,"threads_started":1}
I20260925 22:39:02.477319 22586 raft_consensus.cc:729] T 00000000000000000000000000000000 P c836969096ff45db811a88fa616fd000 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c836969096ff45db811a88fa616fd000, State: Initialized, Role: FOLLOWER
I20260925 22:39:02.477919 22586 consensus_queue.cc:260] T 00000000000000000000000000000000 P c836969096ff45db811a88fa616fd000 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c836969096ff45db811a88fa616fd000" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 43767 } }
I20260925 22:39:02.478098 22586 raft_consensus.cc:388] T 00000000000000000000000000000000 P c836969096ff45db811a88fa616fd000 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:39:02.478195 22586 raft_consensus.cc:482] T 00000000000000000000000000000000 P c836969096ff45db811a88fa616fd000 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:39:02.478303 22586 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c836969096ff45db811a88fa616fd000 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:39:02.479723 22586 raft_consensus.cc:504] T 00000000000000000000000000000000 P c836969096ff45db811a88fa616fd000 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c836969096ff45db811a88fa616fd000" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 43767 } }
I20260925 22:39:02.480136 22586 leader_election.cc:302] T 00000000000000000000000000000000 P c836969096ff45db811a88fa616fd000 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c836969096ff45db811a88fa616fd000; no voters:
I20260925 22:39:02.480545 22586 leader_election.cc:288] T 00000000000000000000000000000000 P c836969096ff45db811a88fa616fd000 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:39:02.480652 22591 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c836969096ff45db811a88fa616fd000 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:39:02.480973 22591 raft_consensus.cc:686] T 00000000000000000000000000000000 P c836969096ff45db811a88fa616fd000 [term 1 LEADER]: Becoming Leader. State: Replica: c836969096ff45db811a88fa616fd000, State: Running, Role: LEADER
I20260925 22:39:02.481390 22591 consensus_queue.cc:237] T 00000000000000000000000000000000 P c836969096ff45db811a88fa616fd000 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c836969096ff45db811a88fa616fd000" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 43767 } }
I20260925 22:39:02.482059 22586 sys_catalog.cc:564] T 00000000000000000000000000000000 P c836969096ff45db811a88fa616fd000 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:39:02.483418 22593 sys_catalog.cc:455] T 00000000000000000000000000000000 P c836969096ff45db811a88fa616fd000 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c836969096ff45db811a88fa616fd000. Latest consensus state: current_term: 1 leader_uuid: "c836969096ff45db811a88fa616fd000" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c836969096ff45db811a88fa616fd000" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 43767 } } }
I20260925 22:39:02.483572 22593 sys_catalog.cc:458] T 00000000000000000000000000000000 P c836969096ff45db811a88fa616fd000 [sys.catalog]: This master's current role is: LEADER
I20260925 22:39:02.484192 22600 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:39:02.484715 22592 sys_catalog.cc:455] T 00000000000000000000000000000000 P c836969096ff45db811a88fa616fd000 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c836969096ff45db811a88fa616fd000" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c836969096ff45db811a88fa616fd000" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 43767 } } }
I20260925 22:39:02.485019 22592 sys_catalog.cc:458] T 00000000000000000000000000000000 P c836969096ff45db811a88fa616fd000 [sys.catalog]: This master's current role is: LEADER
I20260925 22:39:02.487614 22600 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:39:02.493937 22600 catalog_manager.cc:1347] Generated new cluster ID: 1bf9af0db2514738bb75337fbf575332
I20260925 22:39:02.494010 22600 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:39:02.535029 22600 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:39:02.536312 22600 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:39:02.543087 22600 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c836969096ff45db811a88fa616fd000: Generated new TSK 0
I20260925 22:39:02.543987 22600 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.246.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.193.keytab.
Entry for principal kudu/127.6.246.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.193.keytab.
I20260925 22:39:02.587714 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.193:0
--local_ip_for_outbound_sockets=127.6.246.193
--webserver_interface=127.6.246.193
--webserver_port=0
--tserver_master_addrs=127.6.246.254:43767
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.193.keytab
--principal=kudu/127.6.246.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:33735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:39:02.720803 22612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:39:02.721231 22612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:39:02.721290 22612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:39:02.725910 22612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:39:02.726007 22612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:39:02.726078 22612 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.193
I20260925 22:39:02.732062 22612 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:33735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.193.keytab
--principal=kudu/127.6.246.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.246.193
--webserver_port=0
--tserver_master_addrs=127.6.246.254:43767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22612
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.193
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:39:02.733592 22612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:39:02.734939 22612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:39:02.746580 22617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:39:02.746520 22620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:39:02.746522 22618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:39:02.747021 22612 server_base.cc:1034] running on GCE node
I20260925 22:39:02.747540 22612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:39:02.748347 22612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:39:02.749524 22612 hybrid_clock.cc:648] HybridClock initialized: now 1790375942749497 us; error 38 us; skew 500 ppm
Sep 25 22:39:02 dist-test-slave-wjdr krb5kdc[22505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375942, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:39:02.752808 22612 init.cc:377] Logged in from keytab as kudu/127.6.246.193@KRBTEST.COM (short username kudu)
I20260925 22:39:02.753887 22612 webserver.cc:460] Webserver started at http://127.6.246.193:37015/ using document root <none> and password file <none>
I20260925 22:39:02.754571 22612 fs_manager.cc:359] Metadata directory not provided
I20260925 22:39:02.754635 22612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:39:02.754843 22612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:39:02.757378 22612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-0/data/instance:
uuid: "e240ef61505444dd98257c610c9e2702"
format_stamp: "Formatted at 2026-09-25 22:39:02 on dist-test-slave-wjdr"
server_key: "421a1f21ae2191645792feb592e8e95e"
server_key_iv: "40358199eb0614f96ca137c440a729a2"
server_key_version: "encryptionkey@0"
I20260925 22:39:02.757995 22612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-0/wal/instance:
uuid: "e240ef61505444dd98257c610c9e2702"
format_stamp: "Formatted at 2026-09-25 22:39:02 on dist-test-slave-wjdr"
server_key: "421a1f21ae2191645792feb592e8e95e"
server_key_iv: "40358199eb0614f96ca137c440a729a2"
server_key_version: "encryptionkey@0"
I20260925 22:39:02.762451 22612 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:39:02.765267 22627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:39:02.766750 22612 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:39:02.766868 22612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-0/wal
uuid: "e240ef61505444dd98257c610c9e2702"
format_stamp: "Formatted at 2026-09-25 22:39:02 on dist-test-slave-wjdr"
server_key: "421a1f21ae2191645792feb592e8e95e"
server_key_iv: "40358199eb0614f96ca137c440a729a2"
server_key_version: "encryptionkey@0"
I20260925 22:39:02.766971 22612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:39:02.801256 22612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:39:02.804876 22612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:39:02.805060 22612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:39:02.805719 22612 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:39:02.806653 22612 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:39:02.806707 22612 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:39:02.806752 22612 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:39:02.806773 22612 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:39:02.817844 22612 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.193:41485
I20260925 22:39:02.817874 22740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.193:41485 every 8 connection(s)
I20260925 22:39:02.819000 22612 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
I20260925 22:39:02.827518 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 22612
I20260925 22:39:02.827713 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-0/wal/instance
I20260925 22:39:02.828133 7131 external_mini_cluster.cc:1428] Setting key 6830350b840bbb4e7db8d49fb8c2c374
Sep 25 22:39:02 dist-test-slave-wjdr krb5kdc[22505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375942, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.193@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
I20260925 22:39:02.835533 22589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:02.821454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.193:51077 (local address 127.6.246.254:43767)
0925 22:39:02.821681 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 22:39:02.821686 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:39:02.822929 (+ 1243us) server_negotiation.cc:407] Connection header received
0925 22:39:02.823950 (+ 1021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:39:02.823953 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:39:02.824013 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:39:02.824112 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:39:02.825449 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:39:02.826040 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:39:02.827232 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:39:02.827481 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:39:02.827567 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:39:02.830789 (+ 3222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:39:02.830793 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:39:02.830796 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:39:02.830830 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:39:02.832843 (+ 2013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:39:02.833463 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:39:02.833478 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:39:02.833482 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:39:02.833602 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:39:02.833995 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:39:02.833999 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:39:02.834003 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:39:02.834302 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:39:02.834427 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 22:39:02.835046 (+ 619us) server_negotiation.cc:299] Negotiation successful
0925 22:39:02.835225 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260925 22:39:02.836616 22743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:02.821915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43767 (local address 127.6.246.193:51077)
0925 22:39:02.822687 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0925 22:39:02.822772 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0925 22:39:02.823740 (+ 968us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:39:02.824299 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:39:02.824305 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:39:02.824738 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:39:02.825307 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:39:02.825321 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:39:02.826175 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:39:02.826177 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:39:02.827112 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:39:02.827120 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:39:02.827677 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:39:02.827682 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:39:02.827961 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:39:02.828742 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:39:02.828761 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:39:02.830605 (+ 1844us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:39:02.832986 (+ 2381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:39:02.833000 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:39:02.833023 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:39:02.833344 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:39:02.833705 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:39:02.833708 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:39:02.833709 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:39:02.833893 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:39:02.834443 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:39:02.834449 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:39:02.834761 (+ 312us) client_negotiation.cc:769] Sending connection context
0925 22:39:02.834956 (+ 195us) client_negotiation.cc:240] Negotiation successful
0925 22:39:02.835301 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":133,"threads_started":1}
I20260925 22:39:02.838145 22741 heartbeater.cc:347] Connected to a master server at 127.6.246.254:43767
I20260925 22:39:02.838442 22741 heartbeater.cc:464] Registering TS with master...
I20260925 22:39:02.839062 22741 heartbeater.cc:511] Master 127.6.246.254:43767 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.6.246.194@KRBTEST.COM; defaulting to no policy
I20260925 22:39:02.841012 22550 ts_manager.cc:194] Registered new tserver with Master: e240ef61505444dd98257c610c9e2702 (127.6.246.193:41485)
I20260925 22:39:02.842504 22550 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.246.193@KRBTEST.COM'} at 127.6.246.193:51077
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.194.keytab.
Entry for principal kudu/127.6.246.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.194.keytab.
I20260925 22:39:02.860080 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.194:0
--local_ip_for_outbound_sockets=127.6.246.194
--webserver_interface=127.6.246.194
--webserver_port=0
--tserver_master_addrs=127.6.246.254:43767
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.194.keytab
--principal=kudu/127.6.246.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:33735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:39:02.991811 22746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:39:02.992131 22746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:39:02.992179 22746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:39:02.996366 22746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:39:02.996495 22746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:39:02.996598 22746 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.194
I20260925 22:39:03.002439 22746 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:33735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.194.keytab
--principal=kudu/127.6.246.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.246.194
--webserver_port=0
--tserver_master_addrs=127.6.246.254:43767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22746
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.194
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:39:03.003810 22746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:39:03.005129 22746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:39:03.016412 22752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:39:03.016419 22751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:39:03.016427 22754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:39:03.016953 22746 server_base.cc:1034] running on GCE node
I20260925 22:39:03.017396 22746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:39:03.018123 22746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:39:03.019379 22746 hybrid_clock.cc:648] HybridClock initialized: now 1790375943019340 us; error 55 us; skew 500 ppm
Sep 25 22:39:03 dist-test-slave-wjdr krb5kdc[22505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375943, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:39:03.022795 22746 init.cc:377] Logged in from keytab as kudu/127.6.246.194@KRBTEST.COM (short username kudu)
I20260925 22:39:03.023937 22746 webserver.cc:460] Webserver started at http://127.6.246.194:46653/ using document root <none> and password file <none>
I20260925 22:39:03.024721 22746 fs_manager.cc:359] Metadata directory not provided
I20260925 22:39:03.024789 22746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:39:03.025055 22746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:39:03.027490 22746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-1/data/instance:
uuid: "eebff21e25664818b6cefcaa7d386fdc"
format_stamp: "Formatted at 2026-09-25 22:39:03 on dist-test-slave-wjdr"
server_key: "6c0e37dc988440bafb55579286a99c10"
server_key_iv: "436595a353af7510518876d9ff5a5084"
server_key_version: "encryptionkey@0"
I20260925 22:39:03.028139 22746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-1/wal/instance:
uuid: "eebff21e25664818b6cefcaa7d386fdc"
format_stamp: "Formatted at 2026-09-25 22:39:03 on dist-test-slave-wjdr"
server_key: "6c0e37dc988440bafb55579286a99c10"
server_key_iv: "436595a353af7510518876d9ff5a5084"
server_key_version: "encryptionkey@0"
I20260925 22:39:03.032510 22746 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260925 22:39:03.035203 22761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:39:03.036672 22746 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:39:03.036809 22746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-1/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-1/wal
uuid: "eebff21e25664818b6cefcaa7d386fdc"
format_stamp: "Formatted at 2026-09-25 22:39:03 on dist-test-slave-wjdr"
server_key: "6c0e37dc988440bafb55579286a99c10"
server_key_iv: "436595a353af7510518876d9ff5a5084"
server_key_version: "encryptionkey@0"
I20260925 22:39:03.036931 22746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:39:03.068990 22746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:39:03.073091 22746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:39:03.073319 22746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:39:03.074090 22746 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:39:03.075079 22746 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:39:03.075152 22746 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:39:03.075242 22746 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:39:03.075291 22746 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:39:03.085801 22746 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.194:32827
I20260925 22:39:03.085829 22874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.194:32827 every 8 connection(s)
I20260925 22:39:03.086863 22746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
I20260925 22:39:03.088732 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 22746
I20260925 22:39:03.088871 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-1/wal/instance
I20260925 22:39:03.089305 7131 external_mini_cluster.cc:1428] Setting key 46241df6b2ae6a90d17f7db8ac83b63a
Sep 25 22:39:03 dist-test-slave-wjdr krb5kdc[22505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375943, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.194@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.246.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.195@KRBTEST.COM" created.
I20260925 22:39:03.104266 22589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:03.089193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.194:60535 (local address 127.6.246.254:43767)
0925 22:39:03.089390 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 22:39:03.089396 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:39:03.090646 (+ 1250us) server_negotiation.cc:407] Connection header received
0925 22:39:03.091961 (+ 1315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:39:03.091964 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:39:03.092020 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:39:03.092103 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:39:03.093963 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:39:03.094874 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:39:03.096183 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:39:03.096658 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:39:03.096785 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:39:03.099667 (+ 2882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:39:03.099674 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:39:03.099678 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:39:03.099713 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:39:03.101460 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:39:03.102226 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:39:03.102229 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:39:03.102231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:39:03.102315 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:39:03.102661 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:39:03.102663 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:39:03.102665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:39:03.102872 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:39:03.103107 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0925 22:39:03.103762 (+ 655us) server_negotiation.cc:299] Negotiation successful
0925 22:39:03.103942 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 22:39:03.105484 22877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:03.089671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43767 (local address 127.6.246.194:60535)
0925 22:39:03.090484 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0925 22:39:03.090538 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 22:39:03.091720 (+ 1182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:39:03.092346 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:39:03.092358 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:39:03.093003 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:39:03.093806 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:39:03.093823 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:39:03.095038 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:39:03.095041 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:39:03.096044 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:39:03.096055 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:39:03.096828 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:39:03.096832 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:39:03.096998 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:39:03.097755 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:39:03.097775 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:39:03.099455 (+ 1680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:39:03.101594 (+ 2139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:39:03.101600 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:39:03.101617 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:39:03.102054 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:39:03.102406 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:39:03.102408 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:39:03.102409 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:39:03.102569 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:39:03.103125 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:39:03.103135 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:39:03.103442 (+ 307us) client_negotiation.cc:769] Sending connection context
0925 22:39:03.103642 (+ 200us) client_negotiation.cc:240] Negotiation successful
0925 22:39:03.104084 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":132,"threads_started":1}
I20260925 22:39:03.107039 22875 heartbeater.cc:347] Connected to a master server at 127.6.246.254:43767
I20260925 22:39:03.107358 22875 heartbeater.cc:464] Registering TS with master...
I20260925 22:39:03.107992 22875 heartbeater.cc:511] Master 127.6.246.254:43767 requested a full tablet report, sending...
I20260925 22:39:03.109409 22550 ts_manager.cc:194] Registered new tserver with Master: eebff21e25664818b6cefcaa7d386fdc (127.6.246.194:32827)
I20260925 22:39:03.110437 22550 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.246.194@KRBTEST.COM'} at 127.6.246.194:60535
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.195.keytab.
Entry for principal kudu/127.6.246.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.195.keytab.
I20260925 22:39:03.123088 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.195:0
--local_ip_for_outbound_sockets=127.6.246.195
--webserver_interface=127.6.246.195
--webserver_port=0
--tserver_master_addrs=127.6.246.254:43767
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.195.keytab
--principal=kudu/127.6.246.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:33735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:39:03.253852 22880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:39:03.254161 22880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:39:03.254209 22880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:39:03.258445 22880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:39:03.258507 22880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:39:03.258579 22880 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.195
I20260925 22:39:03.264171 22880 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:33735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.195.keytab
--principal=kudu/127.6.246.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.246.195
--webserver_port=0
--tserver_master_addrs=127.6.246.254:43767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22880
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.195
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:39:03.265604 22880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:39:03.266784 22880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 22:39:03.278060 22880 server_base.cc:1034] running on GCE node
W20260925 22:39:03.278020 22888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:39:03.278020 22886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:39:03.278023 22885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:39:03.278812 22880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:39:03.279462 22880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:39:03.280663 22880 hybrid_clock.cc:648] HybridClock initialized: now 1790375943280646 us; error 28 us; skew 500 ppm
Sep 25 22:39:03 dist-test-slave-wjdr krb5kdc[22505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375943, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:39:03.283958 22880 init.cc:377] Logged in from keytab as kudu/127.6.246.195@KRBTEST.COM (short username kudu)
I20260925 22:39:03.285035 22880 webserver.cc:460] Webserver started at http://127.6.246.195:43819/ using document root <none> and password file <none>
I20260925 22:39:03.285709 22880 fs_manager.cc:359] Metadata directory not provided
I20260925 22:39:03.285773 22880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:39:03.286044 22880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:39:03.288331 22880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-2/data/instance:
uuid: "f5beb7dcb6d641b1a4c9217cee5dd91b"
format_stamp: "Formatted at 2026-09-25 22:39:03 on dist-test-slave-wjdr"
server_key: "07a8b4a7b52202ddefcb5a6c904bac9d"
server_key_iv: "92ac20f6012d9991b06ab55746be1658"
server_key_version: "encryptionkey@0"
I20260925 22:39:03.289011 22880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-2/wal/instance:
uuid: "f5beb7dcb6d641b1a4c9217cee5dd91b"
format_stamp: "Formatted at 2026-09-25 22:39:03 on dist-test-slave-wjdr"
server_key: "07a8b4a7b52202ddefcb5a6c904bac9d"
server_key_iv: "92ac20f6012d9991b06ab55746be1658"
server_key_version: "encryptionkey@0"
I20260925 22:39:03.293325 22880 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:39:03.296047 22895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:39:03.297574 22880 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:39:03.297736 22880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-2/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-2/wal
uuid: "f5beb7dcb6d641b1a4c9217cee5dd91b"
format_stamp: "Formatted at 2026-09-25 22:39:03 on dist-test-slave-wjdr"
server_key: "07a8b4a7b52202ddefcb5a6c904bac9d"
server_key_iv: "92ac20f6012d9991b06ab55746be1658"
server_key_version: "encryptionkey@0"
I20260925 22:39:03.297863 22880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:39:03.312638 22880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:39:03.315996 22880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:39:03.316232 22880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:39:03.317004 22880 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:39:03.317958 22880 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:39:03.318039 22880 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:39:03.318113 22880 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:39:03.318167 22880 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:39:03.329008 22880 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.195:41213
I20260925 22:39:03.329041 23008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.195:41213 every 8 connection(s)
I20260925 22:39:03.330071 22880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
I20260925 22:39:03.331081 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 22880
I20260925 22:39:03.331238 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-2/wal/instance
I20260925 22:39:03.331638 7131 external_mini_cluster.cc:1428] Setting key 2d829e8d9f0828f7c5e17046ba6186b7
Sep 25 22:39:03 dist-test-slave-wjdr krb5kdc[22505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375943, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.195@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
I20260925 22:39:03.345777 22589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:03.332428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.195:43913 (local address 127.6.246.254:43767)
0925 22:39:03.332620 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 22:39:03.332624 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:39:03.333947 (+ 1323us) server_negotiation.cc:407] Connection header received
0925 22:39:03.335070 (+ 1123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:39:03.335073 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:39:03.335135 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:39:03.335220 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:39:03.336554 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:39:03.337156 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:39:03.338494 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:39:03.338737 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:39:03.338851 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:39:03.341648 (+ 2797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:39:03.341655 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:39:03.341669 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:39:03.341698 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:39:03.343237 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:39:03.343821 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:39:03.343823 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:39:03.343825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:39:03.343905 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:39:03.344281 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:39:03.344285 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:39:03.344286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:39:03.344592 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:39:03.344714 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 22:39:03.345416 (+ 702us) server_negotiation.cc:299] Negotiation successful
0925 22:39:03.345571 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 22:39:03.346810 23011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:03.332961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43767 (local address 127.6.246.195:43913)
0925 22:39:03.333794 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0925 22:39:03.333839 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 22:39:03.334827 (+ 988us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:39:03.335431 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:39:03.335437 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:39:03.335824 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:39:03.336358 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:39:03.336371 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:39:03.337323 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:39:03.337330 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:39:03.338367 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:39:03.338376 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:39:03.338890 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:39:03.338894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:39:03.339094 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:39:03.339781 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:39:03.339798 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:39:03.341444 (+ 1646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:39:03.343360 (+ 1916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:39:03.343366 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:39:03.343380 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:39:03.343715 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:39:03.344006 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:39:03.344009 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:39:03.344010 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:39:03.344153 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:39:03.344722 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:39:03.344732 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:39:03.345090 (+ 358us) client_negotiation.cc:769] Sending connection context
0925 22:39:03.345305 (+ 215us) client_negotiation.cc:240] Negotiation successful
0925 22:39:03.345724 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":182,"threads_started":1}
I20260925 22:39:03.348120 23009 heartbeater.cc:347] Connected to a master server at 127.6.246.254:43767
I20260925 22:39:03.348395 23009 heartbeater.cc:464] Registering TS with master...
I20260925 22:39:03.348984 23009 heartbeater.cc:511] Master 127.6.246.254:43767 requested a full tablet report, sending...
I20260925 22:39:03.350188 22550 ts_manager.cc:194] Registered new tserver with Master: f5beb7dcb6d641b1a4c9217cee5dd91b (127.6.246.195:41213)
I20260925 22:39:03.350831 22550 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.246.195@KRBTEST.COM'} at 127.6.246.195:43913
I20260925 22:39:03.357937 7131 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:39:03.367697 22589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:03.360175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41504 (local address 127.6.246.254:43767)
0925 22:39:03.360323 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 22:39:03.360327 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:39:03.360547 (+ 220us) server_negotiation.cc:407] Connection header received
0925 22:39:03.360670 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:39:03.360672 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:39:03.360717 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:39:03.360777 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:39:03.361544 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:39:03.362112 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:39:03.362969 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:39:03.363208 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:39:03.363286 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:39:03.364794 (+ 1508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:39:03.364798 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:39:03.364800 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:39:03.364823 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:39:03.366126 (+ 1303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:39:03.366506 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:39:03.366510 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:39:03.366511 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:39:03.366580 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:39:03.366799 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:39:03.366801 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:39:03.366802 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:39:03.367045 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:39:03.367153 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 22:39:03.367366 (+ 213us) server_negotiation.cc:299] Negotiation successful
0925 22:39:03.367461 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 22:39:03.371024 22550 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41504:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:39:03.374876 22550 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:39:03.392441 23022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:03.385973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36792 (local address 127.6.246.194:32827)
0925 22:39:03.387603 (+ 1630us) server_negotiation.cc:206] Beginning negotiation
0925 22:39:03.387615 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 22:39:03.387637 (+ 22us) server_negotiation.cc:407] Connection header received
0925 22:39:03.387726 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:39:03.387734 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:39:03.387904 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:39:03.388071 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:39:03.388327 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:39:03.389222 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:39:03.390894 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:39:03.391622 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:39:03.391734 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:39:03.391919 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0925 22:39:03.392094 (+ 175us) server_negotiation.cc:299] Negotiation successful
0925 22:39:03.392219 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1518,"thread_start_us":144,"threads_started":1}
I20260925 22:39:03.392792 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:03.385798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.194:32827 (local address 127.0.0.1:36792)
0925 22:39:03.386832 (+ 1034us) negotiation.cc:107] Waiting for socket to connect
0925 22:39:03.386901 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0925 22:39:03.387060 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:39:03.388053 (+ 993us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:39:03.388056 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:39:03.388074 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:39:03.388182 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:39:03.388189 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:39:03.389379 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:39:03.389384 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:39:03.390763 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:39:03.390772 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:39:03.391752 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:39:03.391756 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:39:03.391872 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:39:03.391915 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 22:39:03.392040 (+ 125us) client_negotiation.cc:240] Negotiation successful
0925 22:39:03.392325 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":827,"mutex_wait_us":104,"spinlock_wait_cycles":5376,"thread_start_us":118,"threads_started":1}
I20260925 22:39:03.393319 23023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:03.386168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.195:41213 (local address 127.0.0.1:42232)
0925 22:39:03.387612 (+ 1444us) negotiation.cc:107] Waiting for socket to connect
0925 22:39:03.387623 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 22:39:03.387746 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:39:03.388271 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:39:03.388274 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:39:03.388294 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:39:03.388391 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:39:03.388396 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:39:03.389829 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:39:03.389832 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:39:03.391356 (+ 1524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:39:03.391367 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:39:03.392842 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:39:03.392849 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:39:03.392978 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:39:03.393005 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 22:39:03.393078 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 22:39:03.393178 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1388,"mutex_wait_us":86,"thread_start_us":123,"threads_started":1}
I20260925 22:39:03.393923 23024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:03.386621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42232 (local address 127.6.246.195:41213)
0925 22:39:03.387455 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0925 22:39:03.387469 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 22:39:03.387732 (+ 263us) server_negotiation.cc:407] Connection header received
0925 22:39:03.387896 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:39:03.387901 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:39:03.388085 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:39:03.388248 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:39:03.388525 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:39:03.389692 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:39:03.391510 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:39:03.392527 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:39:03.392634 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:39:03.392776 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 22:39:03.393590 (+ 814us) server_negotiation.cc:299] Negotiation successful
0925 22:39:03.393719 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":703,"thread_start_us":190,"threads_started":1}
I20260925 22:39:03.394431 23020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:03.385891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.193:41485 (local address 127.0.0.1:42362)
0925 22:39:03.386834 (+ 943us) negotiation.cc:107] Waiting for socket to connect
0925 22:39:03.386898 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0925 22:39:03.387055 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:39:03.387893 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:39:03.387899 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:39:03.387922 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:39:03.388083 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:39:03.388090 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:39:03.389817 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:39:03.389821 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:39:03.391356 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:39:03.391367 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:39:03.392757 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:39:03.392760 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:39:03.392848 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:39:03.394138 (+ 1290us) client_negotiation.cc:769] Sending connection context
0925 22:39:03.394222 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 22:39:03.394303 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":759,"mutex_wait_us":1471,"spinlock_wait_cycles":17536,"thread_start_us":113,"threads_started":1}
I20260925 22:39:03.394585 23025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:03.386771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42362 (local address 127.6.246.193:41485)
0925 22:39:03.387224 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0925 22:39:03.387233 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 22:39:03.387289 (+ 56us) server_negotiation.cc:407] Connection header received
0925 22:39:03.387382 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:39:03.387391 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:39:03.387662 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:39:03.387828 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:39:03.388243 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:39:03.389658 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:39:03.391502 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:39:03.392362 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:39:03.392515 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:39:03.392693 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0925 22:39:03.394294 (+ 1601us) server_negotiation.cc:299] Negotiation successful
0925 22:39:03.394426 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":129,"threads_started":1}
I20260925 22:39:03.397042 22943 tablet_service.cc:1467] Processing CreateTablet for tablet 9bd784c989c24b2998e6d39c7223856d (DEFAULT_TABLE table=test-table [id=5f0a183b8cdf433197acedc808f39923]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:39:03.397244 22675 tablet_service.cc:1467] Processing CreateTablet for tablet 9bd784c989c24b2998e6d39c7223856d (DEFAULT_TABLE table=test-table [id=5f0a183b8cdf433197acedc808f39923]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:39:03.397037 22809 tablet_service.cc:1467] Processing CreateTablet for tablet 9bd784c989c24b2998e6d39c7223856d (DEFAULT_TABLE table=test-table [id=5f0a183b8cdf433197acedc808f39923]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:39:03.398403 22943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bd784c989c24b2998e6d39c7223856d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:39:03.398396 22675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bd784c989c24b2998e6d39c7223856d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:39:03.398398 22809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bd784c989c24b2998e6d39c7223856d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:39:03.405905 23026 tablet_bootstrap.cc:492] T 9bd784c989c24b2998e6d39c7223856d P f5beb7dcb6d641b1a4c9217cee5dd91b: Bootstrap starting.
I20260925 22:39:03.407258 23028 tablet_bootstrap.cc:492] T 9bd784c989c24b2998e6d39c7223856d P eebff21e25664818b6cefcaa7d386fdc: Bootstrap starting.
I20260925 22:39:03.409241 23026 tablet_bootstrap.cc:654] T 9bd784c989c24b2998e6d39c7223856d P f5beb7dcb6d641b1a4c9217cee5dd91b: Neither blocks nor log segments found. Creating new log.
I20260925 22:39:03.409987 23028 tablet_bootstrap.cc:654] T 9bd784c989c24b2998e6d39c7223856d P eebff21e25664818b6cefcaa7d386fdc: Neither blocks nor log segments found. Creating new log.
I20260925 22:39:03.410285 23026 log.cc:824] T 9bd784c989c24b2998e6d39c7223856d P f5beb7dcb6d641b1a4c9217cee5dd91b: Log is configured to *not* fsync() on all Append() calls
I20260925 22:39:03.410858 23028 log.cc:824] T 9bd784c989c24b2998e6d39c7223856d P eebff21e25664818b6cefcaa7d386fdc: Log is configured to *not* fsync() on all Append() calls
I20260925 22:39:03.411118 23027 tablet_bootstrap.cc:492] T 9bd784c989c24b2998e6d39c7223856d P e240ef61505444dd98257c610c9e2702: Bootstrap starting.
I20260925 22:39:03.412971 23026 tablet_bootstrap.cc:492] T 9bd784c989c24b2998e6d39c7223856d P f5beb7dcb6d641b1a4c9217cee5dd91b: No bootstrap required, opened a new log
I20260925 22:39:03.413213 23026 ts_tablet_manager.cc:1397] T 9bd784c989c24b2998e6d39c7223856d P f5beb7dcb6d641b1a4c9217cee5dd91b: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260925 22:39:03.413452 23027 tablet_bootstrap.cc:654] T 9bd784c989c24b2998e6d39c7223856d P e240ef61505444dd98257c610c9e2702: Neither blocks nor log segments found. Creating new log.
I20260925 22:39:03.414227 23027 log.cc:824] T 9bd784c989c24b2998e6d39c7223856d P e240ef61505444dd98257c610c9e2702: Log is configured to *not* fsync() on all Append() calls
I20260925 22:39:03.415980 23028 tablet_bootstrap.cc:492] T 9bd784c989c24b2998e6d39c7223856d P eebff21e25664818b6cefcaa7d386fdc: No bootstrap required, opened a new log
I20260925 22:39:03.416208 23028 ts_tablet_manager.cc:1397] T 9bd784c989c24b2998e6d39c7223856d P eebff21e25664818b6cefcaa7d386fdc: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260925 22:39:03.416321 23027 tablet_bootstrap.cc:492] T 9bd784c989c24b2998e6d39c7223856d P e240ef61505444dd98257c610c9e2702: No bootstrap required, opened a new log
I20260925 22:39:03.416538 23027 ts_tablet_manager.cc:1397] T 9bd784c989c24b2998e6d39c7223856d P e240ef61505444dd98257c610c9e2702: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 22:39:03.417613 23026 raft_consensus.cc:348] T 9bd784c989c24b2998e6d39c7223856d P f5beb7dcb6d641b1a4c9217cee5dd91b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebff21e25664818b6cefcaa7d386fdc" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 32827 } } peers { permanent_uuid: "e240ef61505444dd98257c610c9e2702" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 41485 } } peers { permanent_uuid: "f5beb7dcb6d641b1a4c9217cee5dd91b" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 41213 } }
I20260925 22:39:03.417933 23026 raft_consensus.cc:374] T 9bd784c989c24b2998e6d39c7223856d P f5beb7dcb6d641b1a4c9217cee5dd91b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:39:03.418004 23026 raft_consensus.cc:729] T 9bd784c989c24b2998e6d39c7223856d P f5beb7dcb6d641b1a4c9217cee5dd91b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5beb7dcb6d641b1a4c9217cee5dd91b, State: Initialized, Role: FOLLOWER
I20260925 22:39:03.418638 23026 consensus_queue.cc:260] T 9bd784c989c24b2998e6d39c7223856d P f5beb7dcb6d641b1a4c9217cee5dd91b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebff21e25664818b6cefcaa7d386fdc" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 32827 } } peers { permanent_uuid: "e240ef61505444dd98257c610c9e2702" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 41485 } } peers { permanent_uuid: "f5beb7dcb6d641b1a4c9217cee5dd91b" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 41213 } }
I20260925 22:39:03.419421 23009 heartbeater.cc:503] Master 127.6.246.254:43767 was elected leader, sending a full tablet report...
I20260925 22:39:03.419795 23026 ts_tablet_manager.cc:1428] T 9bd784c989c24b2998e6d39c7223856d P f5beb7dcb6d641b1a4c9217cee5dd91b: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260925 22:39:03.419917 23028 raft_consensus.cc:348] T 9bd784c989c24b2998e6d39c7223856d P eebff21e25664818b6cefcaa7d386fdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebff21e25664818b6cefcaa7d386fdc" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 32827 } } peers { permanent_uuid: "e240ef61505444dd98257c610c9e2702" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 41485 } } peers { permanent_uuid: "f5beb7dcb6d641b1a4c9217cee5dd91b" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 41213 } }
I20260925 22:39:03.420204 23028 raft_consensus.cc:374] T 9bd784c989c24b2998e6d39c7223856d P eebff21e25664818b6cefcaa7d386fdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:39:03.420250 23028 raft_consensus.cc:729] T 9bd784c989c24b2998e6d39c7223856d P eebff21e25664818b6cefcaa7d386fdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eebff21e25664818b6cefcaa7d386fdc, State: Initialized, Role: FOLLOWER
I20260925 22:39:03.420696 23028 consensus_queue.cc:260] T 9bd784c989c24b2998e6d39c7223856d P eebff21e25664818b6cefcaa7d386fdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebff21e25664818b6cefcaa7d386fdc" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 32827 } } peers { permanent_uuid: "e240ef61505444dd98257c610c9e2702" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 41485 } } peers { permanent_uuid: "f5beb7dcb6d641b1a4c9217cee5dd91b" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 41213 } }
I20260925 22:39:03.421403 22875 heartbeater.cc:503] Master 127.6.246.254:43767 was elected leader, sending a full tablet report...
I20260925 22:39:03.421228 23027 raft_consensus.cc:348] T 9bd784c989c24b2998e6d39c7223856d P e240ef61505444dd98257c610c9e2702 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebff21e25664818b6cefcaa7d386fdc" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 32827 } } peers { permanent_uuid: "e240ef61505444dd98257c610c9e2702" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 41485 } } peers { permanent_uuid: "f5beb7dcb6d641b1a4c9217cee5dd91b" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 41213 } }
I20260925 22:39:03.421476 23027 raft_consensus.cc:374] T 9bd784c989c24b2998e6d39c7223856d P e240ef61505444dd98257c610c9e2702 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:39:03.421519 23027 raft_consensus.cc:729] T 9bd784c989c24b2998e6d39c7223856d P e240ef61505444dd98257c610c9e2702 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e240ef61505444dd98257c610c9e2702, State: Initialized, Role: FOLLOWER
I20260925 22:39:03.421713 23028 ts_tablet_manager.cc:1428] T 9bd784c989c24b2998e6d39c7223856d P eebff21e25664818b6cefcaa7d386fdc: Time spent starting tablet: real 0.005s user 0.003s sys 0.004s
I20260925 22:39:03.421932 23027 consensus_queue.cc:260] T 9bd784c989c24b2998e6d39c7223856d P e240ef61505444dd98257c610c9e2702 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebff21e25664818b6cefcaa7d386fdc" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 32827 } } peers { permanent_uuid: "e240ef61505444dd98257c610c9e2702" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 41485 } } peers { permanent_uuid: "f5beb7dcb6d641b1a4c9217cee5dd91b" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 41213 } }
I20260925 22:39:03.422646 22741 heartbeater.cc:503] Master 127.6.246.254:43767 was elected leader, sending a full tablet report...
I20260925 22:39:03.423019 23027 ts_tablet_manager.cc:1428] T 9bd784c989c24b2998e6d39c7223856d P e240ef61505444dd98257c610c9e2702: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 22:39:03.477411 23033 raft_consensus.cc:482] T 9bd784c989c24b2998e6d39c7223856d P eebff21e25664818b6cefcaa7d386fdc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:39:03.477766 23033 raft_consensus.cc:504] T 9bd784c989c24b2998e6d39c7223856d P eebff21e25664818b6cefcaa7d386fdc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebff21e25664818b6cefcaa7d386fdc" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 32827 } } peers { permanent_uuid: "e240ef61505444dd98257c610c9e2702" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 41485 } } peers { permanent_uuid: "f5beb7dcb6d641b1a4c9217cee5dd91b" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 41213 } }
I20260925 22:39:03.479375 23033 leader_election.cc:288] T 9bd784c989c24b2998e6d39c7223856d P eebff21e25664818b6cefcaa7d386fdc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e240ef61505444dd98257c610c9e2702 (127.6.246.193:41485), f5beb7dcb6d641b1a4c9217cee5dd91b (127.6.246.195:41213)
I20260925 22:39:03.484149 22877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:03.479736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.193:41485 (local address 127.6.246.194:51167)
0925 22:39:03.480003 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0925 22:39:03.480022 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 22:39:03.480155 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:39:03.480529 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:39:03.480532 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:39:03.480555 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:39:03.480736 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:39:03.480745 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:39:03.481679 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:39:03.481682 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:39:03.482732 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:39:03.482750 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:39:03.483656 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:39:03.483661 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:39:03.483783 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:39:03.483811 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 22:39:03.483889 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 22:39:03.483980 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":34}
I20260925 22:39:03.484164 23025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:03.479839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.194:51167 (local address 127.6.246.193:41485)
0925 22:39:03.480004 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 22:39:03.480010 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:39:03.480177 (+ 167us) server_negotiation.cc:407] Connection header received
0925 22:39:03.480268 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:39:03.480273 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:39:03.480352 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:39:03.480448 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:39:03.480880 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:39:03.481555 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:39:03.482882 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:39:03.483511 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:39:03.483645 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:39:03.483709 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 22:39:03.483955 (+ 246us) server_negotiation.cc:299] Negotiation successful
0925 22:39:03.484040 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 22:39:03.484294 23024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:03.479999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.194:53189 (local address 127.6.246.195:41213)
0925 22:39:03.480167 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 22:39:03.480172 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:39:03.480498 (+ 326us) server_negotiation.cc:407] Connection header received
0925 22:39:03.480576 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:39:03.480579 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:39:03.480636 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:39:03.480736 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:39:03.481158 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:39:03.481722 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:39:03.482968 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:39:03.483654 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:39:03.483772 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:39:03.483848 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 22:39:03.484043 (+ 195us) server_negotiation.cc:299] Negotiation successful
0925 22:39:03.484135 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 22:39:03.484750 23035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:03.479809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.195:41213 (local address 127.6.246.194:53189)
0925 22:39:03.480319 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0925 22:39:03.480334 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:39:03.480490 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:39:03.480784 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:39:03.480787 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:39:03.480836 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:39:03.481014 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:39:03.481023 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:39:03.481847 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:39:03.481850 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:39:03.482845 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:39:03.482858 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:39:03.483793 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:39:03.483796 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:39:03.483897 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:39:03.483919 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 22:39:03.484495 (+ 576us) client_negotiation.cc:240] Negotiation successful
0925 22:39:03.484600 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"mutex_wait_us":19,"thread_start_us":178,"threads_started":1}
I20260925 22:39:03.485123 22695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bd784c989c24b2998e6d39c7223856d" candidate_uuid: "eebff21e25664818b6cefcaa7d386fdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e240ef61505444dd98257c610c9e2702" is_pre_election: true
I20260925 22:39:03.485464 22963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bd784c989c24b2998e6d39c7223856d" candidate_uuid: "eebff21e25664818b6cefcaa7d386fdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5beb7dcb6d641b1a4c9217cee5dd91b" is_pre_election: true
I20260925 22:39:03.485535 22695 raft_consensus.cc:2454] T 9bd784c989c24b2998e6d39c7223856d P e240ef61505444dd98257c610c9e2702 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eebff21e25664818b6cefcaa7d386fdc in term 0.
I20260925 22:39:03.485807 22963 raft_consensus.cc:2454] T 9bd784c989c24b2998e6d39c7223856d P f5beb7dcb6d641b1a4c9217cee5dd91b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eebff21e25664818b6cefcaa7d386fdc in term 0.
I20260925 22:39:03.486150 22764 leader_election.cc:302] T 9bd784c989c24b2998e6d39c7223856d P eebff21e25664818b6cefcaa7d386fdc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e240ef61505444dd98257c610c9e2702, eebff21e25664818b6cefcaa7d386fdc; no voters:
I20260925 22:39:03.486498 23033 raft_consensus.cc:2781] T 9bd784c989c24b2998e6d39c7223856d P eebff21e25664818b6cefcaa7d386fdc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:39:03.486619 23033 raft_consensus.cc:482] T 9bd784c989c24b2998e6d39c7223856d P eebff21e25664818b6cefcaa7d386fdc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:39:03.486687 23033 raft_consensus.cc:3037] T 9bd784c989c24b2998e6d39c7223856d P eebff21e25664818b6cefcaa7d386fdc [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:39:03.487948 23033 raft_consensus.cc:504] T 9bd784c989c24b2998e6d39c7223856d P eebff21e25664818b6cefcaa7d386fdc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebff21e25664818b6cefcaa7d386fdc" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 32827 } } peers { permanent_uuid: "e240ef61505444dd98257c610c9e2702" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 41485 } } peers { permanent_uuid: "f5beb7dcb6d641b1a4c9217cee5dd91b" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 41213 } }
I20260925 22:39:03.488375 23033 leader_election.cc:288] T 9bd784c989c24b2998e6d39c7223856d P eebff21e25664818b6cefcaa7d386fdc [CANDIDATE]: Term 1 election: Requested vote from peers e240ef61505444dd98257c610c9e2702 (127.6.246.193:41485), f5beb7dcb6d641b1a4c9217cee5dd91b (127.6.246.195:41213)
I20260925 22:39:03.488870 22695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bd784c989c24b2998e6d39c7223856d" candidate_uuid: "eebff21e25664818b6cefcaa7d386fdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e240ef61505444dd98257c610c9e2702"
I20260925 22:39:03.488859 22963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bd784c989c24b2998e6d39c7223856d" candidate_uuid: "eebff21e25664818b6cefcaa7d386fdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5beb7dcb6d641b1a4c9217cee5dd91b"
I20260925 22:39:03.489018 22963 raft_consensus.cc:3037] T 9bd784c989c24b2998e6d39c7223856d P f5beb7dcb6d641b1a4c9217cee5dd91b [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:39:03.489019 22695 raft_consensus.cc:3037] T 9bd784c989c24b2998e6d39c7223856d P e240ef61505444dd98257c610c9e2702 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:39:03.490603 22963 raft_consensus.cc:2454] T 9bd784c989c24b2998e6d39c7223856d P f5beb7dcb6d641b1a4c9217cee5dd91b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eebff21e25664818b6cefcaa7d386fdc in term 1.
I20260925 22:39:03.490603 22695 raft_consensus.cc:2454] T 9bd784c989c24b2998e6d39c7223856d P e240ef61505444dd98257c610c9e2702 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eebff21e25664818b6cefcaa7d386fdc in term 1.
I20260925 22:39:03.490971 22765 leader_election.cc:302] T 9bd784c989c24b2998e6d39c7223856d P eebff21e25664818b6cefcaa7d386fdc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eebff21e25664818b6cefcaa7d386fdc, f5beb7dcb6d641b1a4c9217cee5dd91b; no voters:
I20260925 22:39:03.491156 23033 raft_consensus.cc:2781] T 9bd784c989c24b2998e6d39c7223856d P eebff21e25664818b6cefcaa7d386fdc [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:39:03.491415 23033 raft_consensus.cc:686] T 9bd784c989c24b2998e6d39c7223856d P eebff21e25664818b6cefcaa7d386fdc [term 1 LEADER]: Becoming Leader. State: Replica: eebff21e25664818b6cefcaa7d386fdc, State: Running, Role: LEADER
I20260925 22:39:03.491827 23033 consensus_queue.cc:237] T 9bd784c989c24b2998e6d39c7223856d P eebff21e25664818b6cefcaa7d386fdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebff21e25664818b6cefcaa7d386fdc" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 32827 } } peers { permanent_uuid: "e240ef61505444dd98257c610c9e2702" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 41485 } } peers { permanent_uuid: "f5beb7dcb6d641b1a4c9217cee5dd91b" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 41213 } }
I20260925 22:39:03.495702 22550 catalog_manager.cc:5495] T 9bd784c989c24b2998e6d39c7223856d P eebff21e25664818b6cefcaa7d386fdc reported cstate change: term changed from 0 to 1, leader changed from <none> to eebff21e25664818b6cefcaa7d386fdc (127.6.246.194). New cstate: current_term: 1 leader_uuid: "eebff21e25664818b6cefcaa7d386fdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebff21e25664818b6cefcaa7d386fdc" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 32827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e240ef61505444dd98257c610c9e2702" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 41485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5beb7dcb6d641b1a4c9217cee5dd91b" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 41213 } health_report { overall_health: UNKNOWN } } }
I20260925 22:39:03.518472 23022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:03.514552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36798 (local address 127.6.246.194:32827)
0925 22:39:03.514710 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 22:39:03.514714 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:39:03.514728 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:39:03.514827 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:39:03.514830 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:39:03.514897 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:39:03.515003 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:39:03.515270 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:39:03.515819 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:39:03.516746 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:39:03.516965 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:39:03.517064 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:39:03.517380 (+ 316us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:39:03.517953 (+ 573us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:39:03.518087 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 22:39:03.518139 (+ 52us) server_negotiation.cc:299] Negotiation successful
0925 22:39:03.518204 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 22:39:03.526156 22695 raft_consensus.cc:1261] T 9bd784c989c24b2998e6d39c7223856d P e240ef61505444dd98257c610c9e2702 [term 1 FOLLOWER]: Refusing update from remote peer eebff21e25664818b6cefcaa7d386fdc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:39:03.526156 22963 raft_consensus.cc:1261] T 9bd784c989c24b2998e6d39c7223856d P f5beb7dcb6d641b1a4c9217cee5dd91b [term 1 FOLLOWER]: Refusing update from remote peer eebff21e25664818b6cefcaa7d386fdc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:39:03.527091 23036 consensus_queue.cc:1035] T 9bd784c989c24b2998e6d39c7223856d P eebff21e25664818b6cefcaa7d386fdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5beb7dcb6d641b1a4c9217cee5dd91b" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 41213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:39:03.527329 23033 consensus_queue.cc:1035] T 9bd784c989c24b2998e6d39c7223856d P eebff21e25664818b6cefcaa7d386fdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "e240ef61505444dd98257c610c9e2702" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 41485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:39:03.547950 22549 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41504:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:39:03.548185 22549 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41504:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:39:03.551306 22549 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c836969096ff45db811a88fa616fd000: Sending DeleteTablet for 3 replicas of tablet 9bd784c989c24b2998e6d39c7223856d
I20260925 22:39:03.552409 22809 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bd784c989c24b2998e6d39c7223856d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:39:03 UTC) from {username='kudu', principal='kudu/127.6.246.254@KRBTEST.COM'} at 127.0.0.1:36792
I20260925 22:39:03.552424 22675 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bd784c989c24b2998e6d39c7223856d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:39:03 UTC) from {username='kudu', principal='kudu/127.6.246.254@KRBTEST.COM'} at 127.0.0.1:42362
I20260925 22:39:03.552536 22943 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bd784c989c24b2998e6d39c7223856d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:39:03 UTC) from {username='kudu', principal='kudu/127.6.246.254@KRBTEST.COM'} at 127.0.0.1:42232
I20260925 22:39:03.553233 23049 tablet_replica.cc:330] T 9bd784c989c24b2998e6d39c7223856d P f5beb7dcb6d641b1a4c9217cee5dd91b: stopping tablet replica
I20260925 22:39:03.553489 23050 tablet_replica.cc:330] T 9bd784c989c24b2998e6d39c7223856d P e240ef61505444dd98257c610c9e2702: stopping tablet replica
I20260925 22:39:03.553520 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 22612
I20260925 22:39:03.553820 23049 raft_consensus.cc:2229] T 9bd784c989c24b2998e6d39c7223856d P f5beb7dcb6d641b1a4c9217cee5dd91b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:39:03.553820 23050 raft_consensus.cc:2229] T 9bd784c989c24b2998e6d39c7223856d P e240ef61505444dd98257c610c9e2702 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:39:03.554354 23051 tablet_replica.cc:330] T 9bd784c989c24b2998e6d39c7223856d P eebff21e25664818b6cefcaa7d386fdc: stopping tablet replica
I20260925 22:39:03.554399 23049 raft_consensus.cc:2258] T 9bd784c989c24b2998e6d39c7223856d P f5beb7dcb6d641b1a4c9217cee5dd91b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:39:03.554706 23051 raft_consensus.cc:2229] T 9bd784c989c24b2998e6d39c7223856d P eebff21e25664818b6cefcaa7d386fdc [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:39:03.555229 23051 raft_consensus.cc:2258] T 9bd784c989c24b2998e6d39c7223856d P eebff21e25664818b6cefcaa7d386fdc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:39:03.556893 23049 ts_tablet_manager.cc:1905] T 9bd784c989c24b2998e6d39c7223856d P f5beb7dcb6d641b1a4c9217cee5dd91b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:39:03.556893 23051 ts_tablet_manager.cc:1905] T 9bd784c989c24b2998e6d39c7223856d P eebff21e25664818b6cefcaa7d386fdc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:39:03.560917 23049 ts_tablet_manager.cc:1918] T 9bd784c989c24b2998e6d39c7223856d P f5beb7dcb6d641b1a4c9217cee5dd91b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:39:03.560917 23051 ts_tablet_manager.cc:1918] T 9bd784c989c24b2998e6d39c7223856d P eebff21e25664818b6cefcaa7d386fdc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:39:03.561051 23051 log.cc:1196] T 9bd784c989c24b2998e6d39c7223856d P eebff21e25664818b6cefcaa7d386fdc: Deleting WAL directory at /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-1/wal/wals/9bd784c989c24b2998e6d39c7223856d
I20260925 22:39:03.561115 23049 log.cc:1196] T 9bd784c989c24b2998e6d39c7223856d P f5beb7dcb6d641b1a4c9217cee5dd91b: Deleting WAL directory at /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-2/wal/wals/9bd784c989c24b2998e6d39c7223856d
I20260925 22:39:03.561470 23051 ts_tablet_manager.cc:1939] T 9bd784c989c24b2998e6d39c7223856d P eebff21e25664818b6cefcaa7d386fdc: Deleting consensus metadata
I20260925 22:39:03.561474 23049 ts_tablet_manager.cc:1939] T 9bd784c989c24b2998e6d39c7223856d P f5beb7dcb6d641b1a4c9217cee5dd91b: Deleting consensus metadata
I20260925 22:39:03.563205 22537 catalog_manager.cc:4841] TS f5beb7dcb6d641b1a4c9217cee5dd91b (127.6.246.195:41213): tablet 9bd784c989c24b2998e6d39c7223856d (table test-table [id=5f0a183b8cdf433197acedc808f39923]) successfully deleted
I20260925 22:39:03.563205 22534 catalog_manager.cc:4841] TS eebff21e25664818b6cefcaa7d386fdc (127.6.246.194:32827): tablet 9bd784c989c24b2998e6d39c7223856d (table test-table [id=5f0a183b8cdf433197acedc808f39923]) successfully deleted
W20260925 22:39:03.564821 22536 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.246.193:41485 (error 108)
I20260925 22:39:03.566001 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 22746
I20260925 22:39:03.566177 23020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:03.565785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.193:41485 (local address 127.0.0.1:42372)
0925 22:39:03.565975 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 22:39:03.566057 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.193:41485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260925 22:39:03.567057 22536 catalog_manager.cc:4584] TS e240ef61505444dd98257c610c9e2702 (127.6.246.193:41485): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9bd784c989c24b2998e6d39c7223856d: Network error: Client connection negotiation failed: client connection to 127.6.246.193:41485: connect: Connection refused (error 111)
I20260925 22:39:03.573683 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 22880
I20260925 22:39:03.580672 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 22518
I20260925 22:39:23.588773 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:43767
--webserver_interface=127.6.246.254
--webserver_port=39397
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab
--principal=kudu/127.6.246.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:33735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.246.254:43767
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:39:23.718722 23053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:39:23.719045 23053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:39:23.719087 23053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:39:23.723455 23053 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 22:39:23.723529 23053 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 22:39:23.723546 23053 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 22:39:23.723559 23053 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:39:23.723578 23053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:39:23.723636 23053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:39:23.723675 23053 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:39:23.723695 23053 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:39:23.729543 23053 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:33735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.6.246.254:43767
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab
--principal=kudu/127.6.246.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:43767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.246.254
--webserver_port=39397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23053
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:39:23.730929 23053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:39:23.732241 23053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 22:39:23.748376 23053 server_base.cc:1034] running on GCE node
W20260925 22:39:23.748477 23062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:39:23.748726 23060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:39:23.748492 23059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:39:23.749591 23053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:39:23.751015 23053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:39:23.752249 23053 hybrid_clock.cc:648] HybridClock initialized: now 1790375963752210 us; error 56 us; skew 500 ppm
Sep 25 22:39:23 dist-test-slave-wjdr krb5kdc[22505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375963, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:39:23.760067 23053 init.cc:377] Logged in from keytab as kudu/127.6.246.254@KRBTEST.COM (short username kudu)
I20260925 22:39:23.761358 23053 webserver.cc:460] Webserver started at http://127.6.246.254:39397/ using document root <none> and password file <none>
I20260925 22:39:23.762120 23053 fs_manager.cc:359] Metadata directory not provided
I20260925 22:39:23.762246 23053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:39:23.767349 23053 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 22:39:23.770264 23069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:39:23.771905 23053 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 22:39:23.772089 23053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/master-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/master-0/wal
uuid: "c836969096ff45db811a88fa616fd000"
format_stamp: "Formatted at 2026-09-25 22:39:02 on dist-test-slave-wjdr"
server_key: "52344f895b6f9d8f0459de73d0663882"
server_key_iv: "970d6b85cc71460be704c5c6758a4c31"
server_key_version: "encryptionkey@0"
I20260925 22:39:23.772924 23053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:39:23.786059 23053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:39:23.790037 23053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:39:23.790302 23053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:39:23.799690 23053 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.254:43767
I20260925 22:39:23.800055 23121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.254:43767 every 8 connection(s)
I20260925 22:39:23.801157 23053 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
I20260925 22:39:23.806741 23122 sys_catalog.cc:263] Verifying existing consensus state
I20260925 22:39:23.807276 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 23053
I20260925 22:39:23.807801 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.193:41485
--local_ip_for_outbound_sockets=127.6.246.193
--tserver_master_addrs=127.6.246.254:43767
--webserver_port=37015
--webserver_interface=127.6.246.193
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.193.keytab
--principal=kudu/127.6.246.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:33735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:39:23.809031 23122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c836969096ff45db811a88fa616fd000: Bootstrap starting.
I20260925 22:39:23.827528 23122 log.cc:824] T 00000000000000000000000000000000 P c836969096ff45db811a88fa616fd000: Log is configured to *not* fsync() on all Append() calls
I20260925 22:39:23.838119 23122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c836969096ff45db811a88fa616fd000: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 22:39:23.838572 23122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c836969096ff45db811a88fa616fd000: Bootstrap complete.
I20260925 22:39:23.843031 23122 raft_consensus.cc:348] T 00000000000000000000000000000000 P c836969096ff45db811a88fa616fd000 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c836969096ff45db811a88fa616fd000" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 43767 } }
I20260925 22:39:23.843500 23122 raft_consensus.cc:729] T 00000000000000000000000000000000 P c836969096ff45db811a88fa616fd000 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c836969096ff45db811a88fa616fd000, State: Initialized, Role: FOLLOWER
I20260925 22:39:23.844106 23122 consensus_queue.cc:260] T 00000000000000000000000000000000 P c836969096ff45db811a88fa616fd000 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c836969096ff45db811a88fa616fd000" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 43767 } }
I20260925 22:39:23.844240 23122 raft_consensus.cc:388] T 00000000000000000000000000000000 P c836969096ff45db811a88fa616fd000 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:39:23.844365 23122 raft_consensus.cc:482] T 00000000000000000000000000000000 P c836969096ff45db811a88fa616fd000 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:39:23.844525 23122 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c836969096ff45db811a88fa616fd000 [term 1 FOLLOWER]: Advancing to term 2
I20260925 22:39:23.846728 23122 raft_consensus.cc:504] T 00000000000000000000000000000000 P c836969096ff45db811a88fa616fd000 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c836969096ff45db811a88fa616fd000" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 43767 } }
I20260925 22:39:23.847121 23122 leader_election.cc:302] T 00000000000000000000000000000000 P c836969096ff45db811a88fa616fd000 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c836969096ff45db811a88fa616fd000; no voters:
I20260925 22:39:23.847493 23122 leader_election.cc:288] T 00000000000000000000000000000000 P c836969096ff45db811a88fa616fd000 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 22:39:23.847579 23126 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c836969096ff45db811a88fa616fd000 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 22:39:23.847838 23126 raft_consensus.cc:686] T 00000000000000000000000000000000 P c836969096ff45db811a88fa616fd000 [term 2 LEADER]: Becoming Leader. State: Replica: c836969096ff45db811a88fa616fd000, State: Running, Role: LEADER
I20260925 22:39:23.848166 23126 consensus_queue.cc:237] T 00000000000000000000000000000000 P c836969096ff45db811a88fa616fd000 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c836969096ff45db811a88fa616fd000" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 43767 } }
I20260925 22:39:23.849001 23122 sys_catalog.cc:564] T 00000000000000000000000000000000 P c836969096ff45db811a88fa616fd000 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:39:23.849679 23127 sys_catalog.cc:455] T 00000000000000000000000000000000 P c836969096ff45db811a88fa616fd000 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c836969096ff45db811a88fa616fd000" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c836969096ff45db811a88fa616fd000" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 43767 } } }
I20260925 22:39:23.849815 23127 sys_catalog.cc:458] T 00000000000000000000000000000000 P c836969096ff45db811a88fa616fd000 [sys.catalog]: This master's current role is: LEADER
I20260925 22:39:23.850183 23128 sys_catalog.cc:455] T 00000000000000000000000000000000 P c836969096ff45db811a88fa616fd000 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c836969096ff45db811a88fa616fd000. Latest consensus state: current_term: 2 leader_uuid: "c836969096ff45db811a88fa616fd000" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c836969096ff45db811a88fa616fd000" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 43767 } } }
I20260925 22:39:23.850278 23128 sys_catalog.cc:458] T 00000000000000000000000000000000 P c836969096ff45db811a88fa616fd000 [sys.catalog]: This master's current role is: LEADER
I20260925 22:39:23.851300 23131 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:39:23.857348 23131 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:39:23.858026 23131 catalog_manager.cc:1259] Loaded cluster ID: 1bf9af0db2514738bb75337fbf575332
I20260925 22:39:23.858096 23131 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:39:23.862156 23131 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:39:23.862921 23131 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c836969096ff45db811a88fa616fd000: Loaded TSK: 0
I20260925 22:39:23.882423 23131 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c836969096ff45db811a88fa616fd000: Generated new TSK 1
I20260925 22:39:23.882742 23131 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 22:39:23.949168 23124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:39:23.949460 23124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:39:23.949508 23124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:39:23.953963 23124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:39:23.954039 23124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:39:23.954137 23124 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.193
I20260925 22:39:23.959981 23124 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:33735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.193.keytab
--principal=kudu/127.6.246.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.193:41485
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.246.193
--webserver_port=37015
--tserver_master_addrs=127.6.246.254:43767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23124
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.193
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:39:23.961269 23124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:39:23.962492 23124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:39:23.973958 23150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:39:23.973951 23149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:39:23.974108 23152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:39:23.974524 23124 server_base.cc:1034] running on GCE node
I20260925 22:39:23.975008 23124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:39:23.975756 23124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:39:23.976985 23124 hybrid_clock.cc:648] HybridClock initialized: now 1790375963976957 us; error 53 us; skew 500 ppm
Sep 25 22:39:23 dist-test-slave-wjdr krb5kdc[22505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375963, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:39:23.980599 23124 init.cc:377] Logged in from keytab as kudu/127.6.246.193@KRBTEST.COM (short username kudu)
I20260925 22:39:23.982050 23124 webserver.cc:460] Webserver started at http://127.6.246.193:37015/ using document root <none> and password file <none>
I20260925 22:39:23.982879 23124 fs_manager.cc:359] Metadata directory not provided
I20260925 22:39:23.982995 23124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:39:23.988886 23124 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 22:39:23.992129 23159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:39:23.993963 23124 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:39:23.994100 23124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-0/wal
uuid: "e240ef61505444dd98257c610c9e2702"
format_stamp: "Formatted at 2026-09-25 22:39:02 on dist-test-slave-wjdr"
server_key: "421a1f21ae2191645792feb592e8e95e"
server_key_iv: "40358199eb0614f96ca137c440a729a2"
server_key_version: "encryptionkey@0"
I20260925 22:39:23.994879 23124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:39:24.027576 23124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:39:24.030736 23124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:39:24.030915 23124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:39:24.031522 23124 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:39:24.032794 23166 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 22:39:24.038249 23124 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 22:39:24.038327 23124 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260925 22:39:24.038393 23124 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 22:39:24.040503 23124 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 22:39:24.040553 23124 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.002s
I20260925 22:39:24.040995 23166 tablet_bootstrap.cc:492] T 9bd784c989c24b2998e6d39c7223856d P e240ef61505444dd98257c610c9e2702: Bootstrap starting.
I20260925 22:39:24.053418 23124 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.193:41485
I20260925 22:39:24.053817 23273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.193:41485 every 8 connection(s)
I20260925 22:39:24.054795 23124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
I20260925 22:39:24.059013 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 23124
I20260925 22:39:24.059774 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.194:32827
--local_ip_for_outbound_sockets=127.6.246.194
--tserver_master_addrs=127.6.246.254:43767
--webserver_port=46653
--webserver_interface=127.6.246.194
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.194.keytab
--principal=kudu/127.6.246.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:33735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:39:24.067557 23166 log.cc:824] T 9bd784c989c24b2998e6d39c7223856d P e240ef61505444dd98257c610c9e2702: Log is configured to *not* fsync() on all Append() calls
Sep 25 22:39:24 dist-test-slave-wjdr krb5kdc[22505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375963, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.193@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
I20260925 22:39:24.077181 23276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:24.057142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43767 (local address 127.6.246.193:34725)
0925 22:39:24.057659 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0925 22:39:24.057714 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 22:39:24.058569 (+ 855us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:39:24.061719 (+ 3150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:39:24.061726 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:39:24.062174 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:39:24.062620 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:39:24.062634 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:39:24.063937 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:39:24.063941 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:39:24.065113 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:39:24.065129 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:39:24.066648 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:39:24.066653 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:39:24.066852 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:39:24.067705 (+ 853us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:39:24.067729 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:39:24.069769 (+ 2040us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:39:24.073113 (+ 3344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:39:24.073119 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:39:24.073132 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:39:24.073502 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:39:24.074034 (+ 532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:39:24.074038 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:39:24.074040 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:39:24.074216 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:39:24.075026 (+ 810us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:39:24.075033 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:39:24.075355 (+ 322us) client_negotiation.cc:769] Sending connection context
0925 22:39:24.075581 (+ 226us) client_negotiation.cc:240] Negotiation successful
0925 22:39:24.075964 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":150,"threads_started":1}
I20260925 22:39:24.077677 23277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:24.057334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.193:34725 (local address 127.6.246.254:43767)
0925 22:39:24.060756 (+ 3422us) server_negotiation.cc:206] Beginning negotiation
0925 22:39:24.060771 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:39:24.060809 (+ 38us) server_negotiation.cc:407] Connection header received
0925 22:39:24.061029 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:39:24.061040 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:39:24.061343 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:39:24.061671 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:39:24.062800 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:39:24.063766 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:39:24.065302 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:39:24.066161 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:39:24.066393 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:39:24.070123 (+ 3730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:39:24.070135 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:39:24.070162 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:39:24.070216 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:39:24.072921 (+ 2705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:39:24.073665 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:39:24.073674 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:39:24.073683 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:39:24.073899 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:39:24.074340 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:39:24.074346 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:39:24.074350 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:39:24.074883 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:39:24.075030 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 22:39:24.075736 (+ 706us) server_negotiation.cc:299] Negotiation successful
0925 22:39:24.076112 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3249,"thread_start_us":199,"threads_started":1}
I20260925 22:39:24.077908 23166 tablet_bootstrap.cc:492] T 9bd784c989c24b2998e6d39c7223856d P e240ef61505444dd98257c610c9e2702: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 22:39:24.078467 23166 tablet_bootstrap.cc:492] T 9bd784c989c24b2998e6d39c7223856d P e240ef61505444dd98257c610c9e2702: Bootstrap complete.
I20260925 22:39:24.079823 23166 ts_tablet_manager.cc:1397] T 9bd784c989c24b2998e6d39c7223856d P e240ef61505444dd98257c610c9e2702: Time spent bootstrapping tablet: real 0.039s user 0.021s sys 0.014s
I20260925 22:39:24.080864 23274 heartbeater.cc:347] Connected to a master server at 127.6.246.254:43767
I20260925 22:39:24.081135 23274 heartbeater.cc:464] Registering TS with master...
I20260925 22:39:24.081807 23274 heartbeater.cc:511] Master 127.6.246.254:43767 requested a full tablet report, sending...
I20260925 22:39:24.082492 23166 raft_consensus.cc:348] T 9bd784c989c24b2998e6d39c7223856d P e240ef61505444dd98257c610c9e2702 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebff21e25664818b6cefcaa7d386fdc" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 32827 } } peers { permanent_uuid: "e240ef61505444dd98257c610c9e2702" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 41485 } } peers { permanent_uuid: "f5beb7dcb6d641b1a4c9217cee5dd91b" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 41213 } }
I20260925 22:39:24.082921 23166 raft_consensus.cc:729] T 9bd784c989c24b2998e6d39c7223856d P e240ef61505444dd98257c610c9e2702 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e240ef61505444dd98257c610c9e2702, State: Initialized, Role: FOLLOWER
I20260925 22:39:24.083461 23166 consensus_queue.cc:260] T 9bd784c989c24b2998e6d39c7223856d P e240ef61505444dd98257c610c9e2702 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebff21e25664818b6cefcaa7d386fdc" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 32827 } } peers { permanent_uuid: "e240ef61505444dd98257c610c9e2702" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 41485 } } peers { permanent_uuid: "f5beb7dcb6d641b1a4c9217cee5dd91b" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 41213 } }
I20260925 22:39:24.084714 23166 ts_tablet_manager.cc:1428] T 9bd784c989c24b2998e6d39c7223856d P e240ef61505444dd98257c610c9e2702: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260925 22:39:24.085084 23086 ts_manager.cc:194] Registered new tserver with Master: e240ef61505444dd98257c610c9e2702 (127.6.246.193:41485)
I20260925 22:39:24.089062 23086 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.246.193@KRBTEST.COM'} at 127.6.246.193:34725
I20260925 22:39:24.092221 23274 heartbeater.cc:503] Master 127.6.246.254:43767 was elected leader, sending a full tablet report...
Sep 25 22:39:24 dist-test-slave-wjdr krb5kdc[22505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375963, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.254@KRBTEST.COM for kudu/127.6.246.193@KRBTEST.COM
I20260925 22:39:24.100376 23282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:24.088522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48860 (local address 127.6.246.193:41485)
0925 22:39:24.088990 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0925 22:39:24.089003 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 22:39:24.089465 (+ 462us) server_negotiation.cc:407] Connection header received
0925 22:39:24.089764 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:39:24.089771 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:39:24.090012 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:39:24.090145 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:39:24.090987 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:39:24.091764 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:39:24.092900 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:39:24.093391 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:39:24.093522 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:39:24.096287 (+ 2765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:39:24.096295 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:39:24.096307 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:39:24.096330 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:39:24.098230 (+ 1900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:39:24.098786 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:39:24.098791 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:39:24.098792 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:39:24.098873 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:39:24.099161 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:39:24.099166 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:39:24.099168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:39:24.099431 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:39:24.099647 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0925 22:39:24.100000 (+ 353us) server_negotiation.cc:299] Negotiation successful
0925 22:39:24.100193 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":149,"threads_started":1}
I20260925 22:39:24.100376 23281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:24.088392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.193:41485 (local address 127.0.0.1:48860)
0925 22:39:24.089305 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0925 22:39:24.089355 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 22:39:24.089578 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:39:24.090386 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:39:24.090391 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:39:24.090749 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:39:24.090863 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:39:24.090869 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:39:24.091890 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:39:24.091895 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:39:24.092786 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:39:24.092799 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:39:24.093550 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:39:24.093555 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:39:24.093716 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:39:24.094390 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:39:24.094406 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:39:24.096130 (+ 1724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:39:24.098366 (+ 2236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:39:24.098377 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:39:24.098384 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:39:24.098655 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:39:24.098973 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:39:24.098976 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:39:24.098979 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:39:24.099069 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:39:24.099623 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:39:24.099631 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:39:24.099765 (+ 134us) client_negotiation.cc:769] Sending connection context
0925 22:39:24.099918 (+ 153us) client_negotiation.cc:240] Negotiation successful
0925 22:39:24.100168 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722,"thread_start_us":163,"threads_started":1}
I20260925 22:39:24.101832 23207 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bd784c989c24b2998e6d39c7223856d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:39:03 UTC) from {username='kudu', principal='kudu/127.6.246.254@KRBTEST.COM'} at 127.0.0.1:48860
I20260925 22:39:24.103001 23283 tablet_replica.cc:330] stopping tablet replica
I20260925 22:39:24.103412 23283 raft_consensus.cc:2229] T 9bd784c989c24b2998e6d39c7223856d P e240ef61505444dd98257c610c9e2702 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:39:24.103724 23283 raft_consensus.cc:2258] T 9bd784c989c24b2998e6d39c7223856d P e240ef61505444dd98257c610c9e2702 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:39:24.105105 23283 ts_tablet_manager.cc:1905] T 9bd784c989c24b2998e6d39c7223856d P e240ef61505444dd98257c610c9e2702: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:39:24.109956 23283 ts_tablet_manager.cc:1918] T 9bd784c989c24b2998e6d39c7223856d P e240ef61505444dd98257c610c9e2702: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:39:24.110064 23283 log.cc:1196] T 9bd784c989c24b2998e6d39c7223856d P e240ef61505444dd98257c610c9e2702: Deleting WAL directory at /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-0/wal/wals/9bd784c989c24b2998e6d39c7223856d
I20260925 22:39:24.110345 23283 ts_tablet_manager.cc:1939] T 9bd784c989c24b2998e6d39c7223856d P e240ef61505444dd98257c610c9e2702: Deleting consensus metadata
I20260925 22:39:24.112522 23072 catalog_manager.cc:4841] TS e240ef61505444dd98257c610c9e2702 (127.6.246.193:41485): tablet 9bd784c989c24b2998e6d39c7223856d (table test-table [id=5f0a183b8cdf433197acedc808f39923]) successfully deleted
W20260925 22:39:24.193675 23278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:39:24.193977 23278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:39:24.194063 23278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:39:24.198058 23278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:39:24.198128 23278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:39:24.198187 23278 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.194
I20260925 22:39:24.203945 23278 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:33735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.194.keytab
--principal=kudu/127.6.246.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.194:32827
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.246.194
--webserver_port=46653
--tserver_master_addrs=127.6.246.254:43767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23278
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.194
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:39:24.205078 23278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:39:24.206106 23278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:39:24.216979 23289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:39:24.217067 23288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:39:24.217043 23291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:39:24.217156 23278 server_base.cc:1034] running on GCE node
I20260925 22:39:24.217756 23278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:39:24.218444 23278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:39:24.219620 23278 hybrid_clock.cc:648] HybridClock initialized: now 1790375964219586 us; error 56 us; skew 500 ppm
Sep 25 22:39:24 dist-test-slave-wjdr krb5kdc[22505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375964, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:39:24.222615 23278 init.cc:377] Logged in from keytab as kudu/127.6.246.194@KRBTEST.COM (short username kudu)
I20260925 22:39:24.223626 23278 webserver.cc:460] Webserver started at http://127.6.246.194:46653/ using document root <none> and password file <none>
I20260925 22:39:24.224226 23278 fs_manager.cc:359] Metadata directory not provided
I20260925 22:39:24.224336 23278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:39:24.228372 23278 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:39:24.230832 23298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:39:24.232254 23278 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:39:24.232421 23278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-1/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-1/wal
uuid: "eebff21e25664818b6cefcaa7d386fdc"
format_stamp: "Formatted at 2026-09-25 22:39:03 on dist-test-slave-wjdr"
server_key: "6c0e37dc988440bafb55579286a99c10"
server_key_iv: "436595a353af7510518876d9ff5a5084"
server_key_version: "encryptionkey@0"
I20260925 22:39:24.233057 23278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:39:24.246505 23278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:39:24.250041 23278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:39:24.250257 23278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:39:24.250870 23278 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:39:24.251751 23278 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:39:24.251823 23278 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:39:24.251899 23278 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:39:24.251950 23278 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:39:24.262558 23278 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.194:32827
I20260925 22:39:24.262622 23411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.194:32827 every 8 connection(s)
I20260925 22:39:24.263662 23278 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
I20260925 22:39:24.270175 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 23278
I20260925 22:39:24.270663 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.195:41213
--local_ip_for_outbound_sockets=127.6.246.195
--tserver_master_addrs=127.6.246.254:43767
--webserver_port=43819
--webserver_interface=127.6.246.195
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.195.keytab
--principal=kudu/127.6.246.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:33735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 22:39:24 dist-test-slave-wjdr krb5kdc[22505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375964, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.194@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
I20260925 22:39:24.280177 23277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:24.265999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.194:39155 (local address 127.6.246.254:43767)
0925 22:39:24.266193 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 22:39:24.266198 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:39:24.267366 (+ 1168us) server_negotiation.cc:407] Connection header received
0925 22:39:24.268361 (+ 995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:39:24.268363 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:39:24.268432 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:39:24.268583 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:39:24.270020 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:39:24.270691 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:39:24.272293 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:39:24.272618 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:39:24.272741 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:39:24.275964 (+ 3223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:39:24.275968 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:39:24.275970 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:39:24.275998 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:39:24.277748 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:39:24.278426 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:39:24.278431 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:39:24.278433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:39:24.278537 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:39:24.278904 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:39:24.278908 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:39:24.278910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:39:24.279119 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:39:24.279223 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 22:39:24.279820 (+ 597us) server_negotiation.cc:299] Negotiation successful
0925 22:39:24.279979 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 22:39:24.281378 23414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:24.266403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43767 (local address 127.6.246.194:39155)
0925 22:39:24.267162 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0925 22:39:24.267213 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 22:39:24.268146 (+ 933us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:39:24.268785 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:39:24.268793 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:39:24.269222 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:39:24.269813 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:39:24.269828 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:39:24.270831 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:39:24.270838 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:39:24.272159 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:39:24.272169 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:39:24.272796 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:39:24.272801 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:39:24.272994 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:39:24.273752 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:39:24.273771 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:39:24.275755 (+ 1984us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:39:24.277876 (+ 2121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:39:24.277883 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:39:24.277914 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:39:24.278313 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:39:24.278638 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:39:24.278641 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:39:24.278643 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:39:24.278798 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:39:24.279241 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:39:24.279249 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:39:24.279548 (+ 299us) client_negotiation.cc:769] Sending connection context
0925 22:39:24.279782 (+ 234us) client_negotiation.cc:240] Negotiation successful
0925 22:39:24.280188 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":129,"threads_started":1}
I20260925 22:39:24.282930 23412 heartbeater.cc:347] Connected to a master server at 127.6.246.254:43767
I20260925 22:39:24.283232 23412 heartbeater.cc:464] Registering TS with master...
I20260925 22:39:24.283828 23412 heartbeater.cc:511] Master 127.6.246.254:43767 requested a full tablet report, sending...
I20260925 22:39:24.285252 23086 ts_manager.cc:194] Registered new tserver with Master: eebff21e25664818b6cefcaa7d386fdc (127.6.246.194:32827)
I20260925 22:39:24.285931 23086 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.246.194@KRBTEST.COM'} at 127.6.246.194:39155
W20260925 22:39:24.384752 23415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:39:24.385174 23415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:39:24.385257 23415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:39:24.390254 23415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:39:24.390369 23415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:39:24.390475 23415 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.195
I20260925 22:39:24.397053 23415 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:33735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.195.keytab
--principal=kudu/127.6.246.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.195:41213
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.246.195
--webserver_port=43819
--tserver_master_addrs=127.6.246.254:43767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23415
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.195
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:39:24.398483 23415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:39:24.399737 23415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:39:24.411195 23420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:39:24.411214 23423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:39:24.411197 23421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:39:24.411474 23415 server_base.cc:1034] running on GCE node
I20260925 22:39:24.412091 23415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:39:24.412909 23415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:39:24.414121 23415 hybrid_clock.cc:648] HybridClock initialized: now 1790375964414087 us; error 48 us; skew 500 ppm
Sep 25 22:39:24 dist-test-slave-wjdr krb5kdc[22505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375964, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:39:24.417281 23415 init.cc:377] Logged in from keytab as kudu/127.6.246.195@KRBTEST.COM (short username kudu)
I20260925 22:39:24.418376 23415 webserver.cc:460] Webserver started at http://127.6.246.195:43819/ using document root <none> and password file <none>
I20260925 22:39:24.419049 23415 fs_manager.cc:359] Metadata directory not provided
I20260925 22:39:24.419129 23415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:39:24.423853 23415 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:39:24.426457 23430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:39:24.428220 23415 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:39:24.428345 23415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-2/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-2/wal
uuid: "f5beb7dcb6d641b1a4c9217cee5dd91b"
format_stamp: "Formatted at 2026-09-25 22:39:03 on dist-test-slave-wjdr"
server_key: "07a8b4a7b52202ddefcb5a6c904bac9d"
server_key_iv: "92ac20f6012d9991b06ab55746be1658"
server_key_version: "encryptionkey@0"
I20260925 22:39:24.429134 23415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:39:24.450611 23415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:39:24.454721 23415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:39:24.454921 23415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:39:24.455627 23415 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:39:24.456756 23415 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:39:24.456807 23415 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:39:24.456881 23415 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:39:24.456919 23415 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:39:24.467350 23415 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.195:41213
I20260925 22:39:24.467373 23543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.195:41213 every 8 connection(s)
I20260925 22:39:24.468614 23415 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
I20260925 22:39:24.478686 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 23415
Sep 25 22:39:24 dist-test-slave-wjdr krb5kdc[22505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375964, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.195@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
I20260925 22:39:24.483841 23277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:24.470814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.195:54179 (local address 127.6.246.254:43767)
0925 22:39:24.470986 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 22:39:24.470990 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:39:24.472173 (+ 1183us) server_negotiation.cc:407] Connection header received
0925 22:39:24.473221 (+ 1048us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:39:24.473223 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:39:24.473277 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:39:24.473354 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:39:24.474692 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:39:24.475288 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:39:24.476475 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:39:24.476691 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:39:24.476769 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:39:24.479838 (+ 3069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:39:24.479842 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:39:24.479844 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:39:24.479864 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:39:24.481665 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:39:24.482215 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:39:24.482218 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:39:24.482219 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:39:24.482293 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:39:24.482666 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:39:24.482668 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:39:24.482669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:39:24.482839 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:39:24.482916 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 22:39:24.483558 (+ 642us) server_negotiation.cc:299] Negotiation successful
0925 22:39:24.483660 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 22:39:24.485033 23546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:24.471260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43767 (local address 127.6.246.195:54179)
0925 22:39:24.471989 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0925 22:39:24.472033 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 22:39:24.472996 (+ 963us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:39:24.473541 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:39:24.473548 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:39:24.473983 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:39:24.474555 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:39:24.474570 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:39:24.475403 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:39:24.475406 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:39:24.476356 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:39:24.476365 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:39:24.476828 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:39:24.476866 (+ 38us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:39:24.477048 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:39:24.477748 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:39:24.477765 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:39:24.479655 (+ 1890us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:39:24.481781 (+ 2126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:39:24.481787 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:39:24.481802 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:39:24.482119 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:39:24.482380 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:39:24.482383 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:39:24.482385 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:39:24.482570 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:39:24.482939 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:39:24.482946 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:39:24.483279 (+ 333us) client_negotiation.cc:769] Sending connection context
0925 22:39:24.483503 (+ 224us) client_negotiation.cc:240] Negotiation successful
0925 22:39:24.483925 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":125,"threads_started":1}
I20260925 22:39:24.486537 23544 heartbeater.cc:347] Connected to a master server at 127.6.246.254:43767
I20260925 22:39:24.486874 23544 heartbeater.cc:464] Registering TS with master...
I20260925 22:39:24.487620 23544 heartbeater.cc:511] Master 127.6.246.254:43767 requested a full tablet report, sending...
I20260925 22:39:24.488323 23547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:24.479463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50886 (local address 127.6.246.254:43767)
0925 22:39:24.479737 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0925 22:39:24.479741 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:39:24.479977 (+ 236us) server_negotiation.cc:407] Connection header received
0925 22:39:24.480100 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:39:24.480105 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:39:24.480163 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:39:24.480267 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:39:24.481221 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:39:24.482000 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:39:24.483109 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:39:24.483330 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:39:24.483403 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:39:24.484941 (+ 1538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:39:24.484944 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:39:24.484946 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:39:24.484967 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:39:24.486685 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:39:24.487152 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:39:24.487154 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:39:24.487156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:39:24.487228 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:39:24.487498 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:39:24.487500 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:39:24.487501 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:39:24.487667 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:39:24.487866 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0925 22:39:24.488068 (+ 202us) server_negotiation.cc:299] Negotiation successful
0925 22:39:24.488168 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":122,"threads_started":1}
I20260925 22:39:24.489022 23082 ts_manager.cc:194] Registered new tserver with Master: f5beb7dcb6d641b1a4c9217cee5dd91b (127.6.246.195:41213)
I20260925 22:39:24.489686 23082 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.246.195@KRBTEST.COM'} at 127.6.246.195:54179
I20260925 22:39:24.490332 7131 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:39:24.494534 23547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:24.491011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50894 (local address 127.6.246.254:43767)
0925 22:39:24.491132 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 22:39:24.491137 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:39:24.491442 (+ 305us) server_negotiation.cc:407] Connection header received
0925 22:39:24.491611 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:39:24.491614 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:39:24.491713 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:39:24.491796 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:39:24.492042 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:39:24.492647 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:39:24.493482 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:39:24.493701 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:39:24.493789 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:39:24.494031 (+ 242us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:39:24.494182 (+ 151us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0925 22:39:24.494396 (+ 214us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50894: token expired
Metrics: {"server-negotiator.queue_time_us":43}
W20260925 22:39:24.494601 23547 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50894: token expired
W20260925 22:39:24.494990 23013 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.6.246.254:43767)
I20260925 22:39:24.503417 23547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:24.495363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50900 (local address 127.6.246.254:43767)
0925 22:39:24.495482 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0925 22:39:24.495485 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:39:24.495497 (+ 12us) server_negotiation.cc:407] Connection header received
0925 22:39:24.495597 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:39:24.495599 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:39:24.495632 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:39:24.495694 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:39:24.496263 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:39:24.496792 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:39:24.497568 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:39:24.497776 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:39:24.497851 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:39:24.499424 (+ 1573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:39:24.499430 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:39:24.499434 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:39:24.499460 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:39:24.501566 (+ 2106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:39:24.502070 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:39:24.502074 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:39:24.502076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:39:24.502210 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:39:24.502521 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:39:24.502524 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:39:24.502526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:39:24.502759 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:39:24.502907 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 22:39:24.503115 (+ 208us) server_negotiation.cc:299] Negotiation successful
0925 22:39:24.503215 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 22:39:24.506521 23086 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50900:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:39:24.510732 23086 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:39:24.519781 23207 tablet_service.cc:1467] Processing CreateTablet for tablet bb84ba73ab9344b8b60560450f51c9b2 (DEFAULT_TABLE table=test-table [id=61182d6ae946411db82fc4713fa1d394]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:39:24.520659 23207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb84ba73ab9344b8b60560450f51c9b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:39:24.524109 23166 tablet_bootstrap.cc:492] T bb84ba73ab9344b8b60560450f51c9b2 P e240ef61505444dd98257c610c9e2702: Bootstrap starting.
I20260925 22:39:24.524590 23281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:24.518592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.195:41213 (local address 127.0.0.1:38864)
0925 22:39:24.518732 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 22:39:24.518747 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:39:24.518852 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:39:24.520258 (+ 1406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:39:24.520264 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:39:24.520283 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:39:24.520428 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:39:24.520436 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:39:24.521808 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:39:24.521815 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:39:24.523106 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:39:24.523115 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:39:24.524079 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:39:24.524082 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:39:24.524194 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:39:24.524256 (+ 62us) client_negotiation.cc:769] Sending connection context
0925 22:39:24.524312 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 22:39:24.524400 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260925 22:39:24.524845 23554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:24.518981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.194:32827 (local address 127.0.0.1:34154)
0925 22:39:24.519266 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0925 22:39:24.519280 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:39:24.519372 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:39:24.520672 (+ 1300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:39:24.520675 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:39:24.520689 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:39:24.520788 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:39:24.520794 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:39:24.522148 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:39:24.522152 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:39:24.523243 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:39:24.523250 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:39:24.524439 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:39:24.524442 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:39:24.524556 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:39:24.524573 (+ 17us) client_negotiation.cc:769] Sending connection context
0925 22:39:24.524621 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 22:39:24.524713 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":124,"threads_started":1}
I20260925 22:39:24.525209 23166 tablet_bootstrap.cc:654] T bb84ba73ab9344b8b60560450f51c9b2 P e240ef61505444dd98257c610c9e2702: Neither blocks nor log segments found. Creating new log.
I20260925 22:39:24.525336 23553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:24.518889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38864 (local address 127.6.246.195:41213)
0925 22:39:24.519595 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0925 22:39:24.519613 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 22:39:24.519642 (+ 29us) server_negotiation.cc:407] Connection header received
0925 22:39:24.519742 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:39:24.519750 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:39:24.520048 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:39:24.520653 (+ 605us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:39:24.520850 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:39:24.521692 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:39:24.523252 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:39:24.523921 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:39:24.524780 (+ 859us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:39:24.524946 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 22:39:24.525002 (+ 56us) server_negotiation.cc:299] Negotiation successful
0925 22:39:24.525153 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":196,"threads_started":1}
I20260925 22:39:24.525553 23555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:24.519438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34154 (local address 127.6.246.194:32827)
0925 22:39:24.519961 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0925 22:39:24.519974 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 22:39:24.520191 (+ 217us) server_negotiation.cc:407] Connection header received
0925 22:39:24.520288 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:39:24.520293 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:39:24.520504 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:39:24.520681 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:39:24.521001 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:39:24.522022 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:39:24.523350 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:39:24.524314 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:39:24.525013 (+ 699us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:39:24.525190 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0925 22:39:24.525244 (+ 54us) server_negotiation.cc:299] Negotiation successful
0925 22:39:24.525365 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":141,"threads_started":1}
I20260925 22:39:24.527046 23166 tablet_bootstrap.cc:492] T bb84ba73ab9344b8b60560450f51c9b2 P e240ef61505444dd98257c610c9e2702: No bootstrap required, opened a new log
I20260925 22:39:24.527155 23166 ts_tablet_manager.cc:1397] T bb84ba73ab9344b8b60560450f51c9b2 P e240ef61505444dd98257c610c9e2702: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20260925 22:39:24.527652 23166 raft_consensus.cc:348] T bb84ba73ab9344b8b60560450f51c9b2 P e240ef61505444dd98257c610c9e2702 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5beb7dcb6d641b1a4c9217cee5dd91b" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 41213 } } peers { permanent_uuid: "eebff21e25664818b6cefcaa7d386fdc" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 32827 } } peers { permanent_uuid: "e240ef61505444dd98257c610c9e2702" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 41485 } }
I20260925 22:39:24.527777 23166 raft_consensus.cc:374] T bb84ba73ab9344b8b60560450f51c9b2 P e240ef61505444dd98257c610c9e2702 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:39:24.527813 23166 raft_consensus.cc:729] T bb84ba73ab9344b8b60560450f51c9b2 P e240ef61505444dd98257c610c9e2702 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e240ef61505444dd98257c610c9e2702, State: Initialized, Role: FOLLOWER
I20260925 22:39:24.527954 23166 consensus_queue.cc:260] T bb84ba73ab9344b8b60560450f51c9b2 P e240ef61505444dd98257c610c9e2702 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5beb7dcb6d641b1a4c9217cee5dd91b" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 41213 } } peers { permanent_uuid: "eebff21e25664818b6cefcaa7d386fdc" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 32827 } } peers { permanent_uuid: "e240ef61505444dd98257c610c9e2702" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 41485 } }
I20260925 22:39:24.528244 23166 ts_tablet_manager.cc:1428] T bb84ba73ab9344b8b60560450f51c9b2 P e240ef61505444dd98257c610c9e2702: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:39:24.529002 23478 tablet_service.cc:1467] Processing CreateTablet for tablet bb84ba73ab9344b8b60560450f51c9b2 (DEFAULT_TABLE table=test-table [id=61182d6ae946411db82fc4713fa1d394]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:39:24.528996 23346 tablet_service.cc:1467] Processing CreateTablet for tablet bb84ba73ab9344b8b60560450f51c9b2 (DEFAULT_TABLE table=test-table [id=61182d6ae946411db82fc4713fa1d394]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:39:24.530263 23346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb84ba73ab9344b8b60560450f51c9b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:39:24.530277 23478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb84ba73ab9344b8b60560450f51c9b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:39:24.538533 23558 tablet_bootstrap.cc:492] T bb84ba73ab9344b8b60560450f51c9b2 P f5beb7dcb6d641b1a4c9217cee5dd91b: Bootstrap starting.
I20260925 22:39:24.538532 23557 tablet_bootstrap.cc:492] T bb84ba73ab9344b8b60560450f51c9b2 P eebff21e25664818b6cefcaa7d386fdc: Bootstrap starting.
I20260925 22:39:24.541579 23558 tablet_bootstrap.cc:654] T bb84ba73ab9344b8b60560450f51c9b2 P f5beb7dcb6d641b1a4c9217cee5dd91b: Neither blocks nor log segments found. Creating new log.
I20260925 22:39:24.541579 23557 tablet_bootstrap.cc:654] T bb84ba73ab9344b8b60560450f51c9b2 P eebff21e25664818b6cefcaa7d386fdc: Neither blocks nor log segments found. Creating new log.
I20260925 22:39:24.542464 23557 log.cc:824] T bb84ba73ab9344b8b60560450f51c9b2 P eebff21e25664818b6cefcaa7d386fdc: Log is configured to *not* fsync() on all Append() calls
I20260925 22:39:24.542465 23558 log.cc:824] T bb84ba73ab9344b8b60560450f51c9b2 P f5beb7dcb6d641b1a4c9217cee5dd91b: Log is configured to *not* fsync() on all Append() calls
I20260925 22:39:24.544694 23557 tablet_bootstrap.cc:492] T bb84ba73ab9344b8b60560450f51c9b2 P eebff21e25664818b6cefcaa7d386fdc: No bootstrap required, opened a new log
I20260925 22:39:24.544697 23558 tablet_bootstrap.cc:492] T bb84ba73ab9344b8b60560450f51c9b2 P f5beb7dcb6d641b1a4c9217cee5dd91b: No bootstrap required, opened a new log
I20260925 22:39:24.544925 23557 ts_tablet_manager.cc:1397] T bb84ba73ab9344b8b60560450f51c9b2 P eebff21e25664818b6cefcaa7d386fdc: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260925 22:39:24.544930 23558 ts_tablet_manager.cc:1397] T bb84ba73ab9344b8b60560450f51c9b2 P f5beb7dcb6d641b1a4c9217cee5dd91b: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260925 22:39:24.548601 23557 raft_consensus.cc:348] T bb84ba73ab9344b8b60560450f51c9b2 P eebff21e25664818b6cefcaa7d386fdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5beb7dcb6d641b1a4c9217cee5dd91b" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 41213 } } peers { permanent_uuid: "eebff21e25664818b6cefcaa7d386fdc" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 32827 } } peers { permanent_uuid: "e240ef61505444dd98257c610c9e2702" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 41485 } }
I20260925 22:39:24.548602 23558 raft_consensus.cc:348] T bb84ba73ab9344b8b60560450f51c9b2 P f5beb7dcb6d641b1a4c9217cee5dd91b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5beb7dcb6d641b1a4c9217cee5dd91b" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 41213 } } peers { permanent_uuid: "eebff21e25664818b6cefcaa7d386fdc" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 32827 } } peers { permanent_uuid: "e240ef61505444dd98257c610c9e2702" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 41485 } }
I20260925 22:39:24.548902 23557 raft_consensus.cc:374] T bb84ba73ab9344b8b60560450f51c9b2 P eebff21e25664818b6cefcaa7d386fdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:39:24.548898 23558 raft_consensus.cc:374] T bb84ba73ab9344b8b60560450f51c9b2 P f5beb7dcb6d641b1a4c9217cee5dd91b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:39:24.548990 23557 raft_consensus.cc:729] T bb84ba73ab9344b8b60560450f51c9b2 P eebff21e25664818b6cefcaa7d386fdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eebff21e25664818b6cefcaa7d386fdc, State: Initialized, Role: FOLLOWER
I20260925 22:39:24.549000 23558 raft_consensus.cc:729] T bb84ba73ab9344b8b60560450f51c9b2 P f5beb7dcb6d641b1a4c9217cee5dd91b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5beb7dcb6d641b1a4c9217cee5dd91b, State: Initialized, Role: FOLLOWER
I20260925 22:39:24.549556 23558 consensus_queue.cc:260] T bb84ba73ab9344b8b60560450f51c9b2 P f5beb7dcb6d641b1a4c9217cee5dd91b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5beb7dcb6d641b1a4c9217cee5dd91b" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 41213 } } peers { permanent_uuid: "eebff21e25664818b6cefcaa7d386fdc" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 32827 } } peers { permanent_uuid: "e240ef61505444dd98257c610c9e2702" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 41485 } }
I20260925 22:39:24.549587 23557 consensus_queue.cc:260] T bb84ba73ab9344b8b60560450f51c9b2 P eebff21e25664818b6cefcaa7d386fdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5beb7dcb6d641b1a4c9217cee5dd91b" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 41213 } } peers { permanent_uuid: "eebff21e25664818b6cefcaa7d386fdc" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 32827 } } peers { permanent_uuid: "e240ef61505444dd98257c610c9e2702" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 41485 } }
I20260925 22:39:24.550346 23412 heartbeater.cc:503] Master 127.6.246.254:43767 was elected leader, sending a full tablet report...
I20260925 22:39:24.550346 23544 heartbeater.cc:503] Master 127.6.246.254:43767 was elected leader, sending a full tablet report...
I20260925 22:39:24.550844 23558 ts_tablet_manager.cc:1428] T bb84ba73ab9344b8b60560450f51c9b2 P f5beb7dcb6d641b1a4c9217cee5dd91b: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260925 22:39:24.552026 23557 ts_tablet_manager.cc:1428] T bb84ba73ab9344b8b60560450f51c9b2 P eebff21e25664818b6cefcaa7d386fdc: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260925 22:39:24.556420 23275 tablet.cc:2369] T bb84ba73ab9344b8b60560450f51c9b2 P e240ef61505444dd98257c610c9e2702: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:39:24.680213 23562 raft_consensus.cc:482] T bb84ba73ab9344b8b60560450f51c9b2 P eebff21e25664818b6cefcaa7d386fdc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:39:24.680590 23562 raft_consensus.cc:504] T bb84ba73ab9344b8b60560450f51c9b2 P eebff21e25664818b6cefcaa7d386fdc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5beb7dcb6d641b1a4c9217cee5dd91b" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 41213 } } peers { permanent_uuid: "eebff21e25664818b6cefcaa7d386fdc" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 32827 } } peers { permanent_uuid: "e240ef61505444dd98257c610c9e2702" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 41485 } }
I20260925 22:39:24.682232 23562 leader_election.cc:288] T bb84ba73ab9344b8b60560450f51c9b2 P eebff21e25664818b6cefcaa7d386fdc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f5beb7dcb6d641b1a4c9217cee5dd91b (127.6.246.195:41213), e240ef61505444dd98257c610c9e2702 (127.6.246.193:41485)
I20260925 22:39:24.688091 23553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:24.682696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.194:34399 (local address 127.6.246.195:41213)
0925 22:39:24.682861 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 22:39:24.682868 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:39:24.683244 (+ 376us) server_negotiation.cc:407] Connection header received
0925 22:39:24.683393 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:39:24.683400 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:39:24.683471 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:39:24.683548 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:39:24.683968 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:39:24.684854 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:39:24.686639 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:39:24.687255 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:39:24.687399 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:39:24.687484 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 22:39:24.687821 (+ 337us) server_negotiation.cc:299] Negotiation successful
0925 22:39:24.687921 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 22:39:24.688098 23414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:24.682577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.193:41485 (local address 127.6.246.194:42989)
0925 22:39:24.682829 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0925 22:39:24.682849 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 22:39:24.682984 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:39:24.683560 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:39:24.683565 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:39:24.683594 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:39:24.683815 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:39:24.683825 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:39:24.685013 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:39:24.685017 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:39:24.686506 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:39:24.686512 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:39:24.687499 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:39:24.687503 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:39:24.687636 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:39:24.687681 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 22:39:24.687782 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 22:39:24.687899 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":139}
I20260925 22:39:24.688113 23564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:24.682578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.195:41213 (local address 127.6.246.194:34399)
0925 22:39:24.683098 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0925 22:39:24.683118 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 22:39:24.683265 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:39:24.683578 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:39:24.683580 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:39:24.683606 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:39:24.683827 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:39:24.683833 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:39:24.685039 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:39:24.685041 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:39:24.686499 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:39:24.686512 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:39:24.687398 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:39:24.687401 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:39:24.687563 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:39:24.687684 (+ 121us) client_negotiation.cc:769] Sending connection context
0925 22:39:24.687803 (+ 119us) client_negotiation.cc:240] Negotiation successful
0925 22:39:24.687898 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"mutex_wait_us":291,"spinlock_wait_cycles":13696,"thread_start_us":153,"threads_started":1}
I20260925 22:39:24.688092 23563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:24.682673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.194:42989 (local address 127.6.246.193:41485)
0925 22:39:24.683105 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0925 22:39:24.683111 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:39:24.683129 (+ 18us) server_negotiation.cc:407] Connection header received
0925 22:39:24.683236 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:39:24.683239 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:39:24.683329 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:39:24.683475 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:39:24.683963 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:39:24.684848 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:39:24.686633 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:39:24.687373 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:39:24.687482 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:39:24.687644 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 22:39:24.687803 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 22:39:24.687921 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":141,"threads_started":1}
I20260925 22:39:24.689147 23498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb84ba73ab9344b8b60560450f51c9b2" candidate_uuid: "eebff21e25664818b6cefcaa7d386fdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5beb7dcb6d641b1a4c9217cee5dd91b" is_pre_election: true
I20260925 22:39:24.689332 23220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb84ba73ab9344b8b60560450f51c9b2" candidate_uuid: "eebff21e25664818b6cefcaa7d386fdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e240ef61505444dd98257c610c9e2702" is_pre_election: true
I20260925 22:39:24.689597 23220 raft_consensus.cc:2454] T bb84ba73ab9344b8b60560450f51c9b2 P e240ef61505444dd98257c610c9e2702 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eebff21e25664818b6cefcaa7d386fdc in term 0.
I20260925 22:39:24.689599 23498 raft_consensus.cc:2454] T bb84ba73ab9344b8b60560450f51c9b2 P f5beb7dcb6d641b1a4c9217cee5dd91b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eebff21e25664818b6cefcaa7d386fdc in term 0.
I20260925 22:39:24.690193 23302 leader_election.cc:302] T bb84ba73ab9344b8b60560450f51c9b2 P eebff21e25664818b6cefcaa7d386fdc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: e240ef61505444dd98257c610c9e2702, eebff21e25664818b6cefcaa7d386fdc, f5beb7dcb6d641b1a4c9217cee5dd91b; no voters:
I20260925 22:39:24.690557 23562 raft_consensus.cc:2781] T bb84ba73ab9344b8b60560450f51c9b2 P eebff21e25664818b6cefcaa7d386fdc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:39:24.690665 23562 raft_consensus.cc:482] T bb84ba73ab9344b8b60560450f51c9b2 P eebff21e25664818b6cefcaa7d386fdc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:39:24.690749 23562 raft_consensus.cc:3037] T bb84ba73ab9344b8b60560450f51c9b2 P eebff21e25664818b6cefcaa7d386fdc [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:39:24.692157 23562 raft_consensus.cc:504] T bb84ba73ab9344b8b60560450f51c9b2 P eebff21e25664818b6cefcaa7d386fdc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5beb7dcb6d641b1a4c9217cee5dd91b" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 41213 } } peers { permanent_uuid: "eebff21e25664818b6cefcaa7d386fdc" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 32827 } } peers { permanent_uuid: "e240ef61505444dd98257c610c9e2702" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 41485 } }
I20260925 22:39:24.692639 23562 leader_election.cc:288] T bb84ba73ab9344b8b60560450f51c9b2 P eebff21e25664818b6cefcaa7d386fdc [CANDIDATE]: Term 1 election: Requested vote from peers f5beb7dcb6d641b1a4c9217cee5dd91b (127.6.246.195:41213), e240ef61505444dd98257c610c9e2702 (127.6.246.193:41485)
I20260925 22:39:24.693084 23220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb84ba73ab9344b8b60560450f51c9b2" candidate_uuid: "eebff21e25664818b6cefcaa7d386fdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e240ef61505444dd98257c610c9e2702"
I20260925 22:39:24.693090 23498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb84ba73ab9344b8b60560450f51c9b2" candidate_uuid: "eebff21e25664818b6cefcaa7d386fdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5beb7dcb6d641b1a4c9217cee5dd91b"
I20260925 22:39:24.693238 23220 raft_consensus.cc:3037] T bb84ba73ab9344b8b60560450f51c9b2 P e240ef61505444dd98257c610c9e2702 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:39:24.693269 23498 raft_consensus.cc:3037] T bb84ba73ab9344b8b60560450f51c9b2 P f5beb7dcb6d641b1a4c9217cee5dd91b [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:39:24.694384 23220 raft_consensus.cc:2454] T bb84ba73ab9344b8b60560450f51c9b2 P e240ef61505444dd98257c610c9e2702 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eebff21e25664818b6cefcaa7d386fdc in term 1.
I20260925 22:39:24.694478 23498 raft_consensus.cc:2454] T bb84ba73ab9344b8b60560450f51c9b2 P f5beb7dcb6d641b1a4c9217cee5dd91b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eebff21e25664818b6cefcaa7d386fdc in term 1.
I20260925 22:39:24.694797 23301 leader_election.cc:302] T bb84ba73ab9344b8b60560450f51c9b2 P eebff21e25664818b6cefcaa7d386fdc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: e240ef61505444dd98257c610c9e2702, eebff21e25664818b6cefcaa7d386fdc, f5beb7dcb6d641b1a4c9217cee5dd91b; no voters:
I20260925 22:39:24.694981 23562 raft_consensus.cc:2781] T bb84ba73ab9344b8b60560450f51c9b2 P eebff21e25664818b6cefcaa7d386fdc [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:39:24.695261 23562 raft_consensus.cc:686] T bb84ba73ab9344b8b60560450f51c9b2 P eebff21e25664818b6cefcaa7d386fdc [term 1 LEADER]: Becoming Leader. State: Replica: eebff21e25664818b6cefcaa7d386fdc, State: Running, Role: LEADER
I20260925 22:39:24.695674 23562 consensus_queue.cc:237] T bb84ba73ab9344b8b60560450f51c9b2 P eebff21e25664818b6cefcaa7d386fdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5beb7dcb6d641b1a4c9217cee5dd91b" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 41213 } } peers { permanent_uuid: "eebff21e25664818b6cefcaa7d386fdc" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 32827 } } peers { permanent_uuid: "e240ef61505444dd98257c610c9e2702" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 41485 } }
I20260925 22:39:24.699671 23082 catalog_manager.cc:5495] T bb84ba73ab9344b8b60560450f51c9b2 P eebff21e25664818b6cefcaa7d386fdc reported cstate change: term changed from 0 to 1, leader changed from <none> to eebff21e25664818b6cefcaa7d386fdc (127.6.246.194). New cstate: current_term: 1 leader_uuid: "eebff21e25664818b6cefcaa7d386fdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5beb7dcb6d641b1a4c9217cee5dd91b" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 41213 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eebff21e25664818b6cefcaa7d386fdc" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 32827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e240ef61505444dd98257c610c9e2702" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 41485 } health_report { overall_health: UNKNOWN } } }
I20260925 22:39:24.717236 23555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:24.713451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34156 (local address 127.6.246.194:32827)
0925 22:39:24.713605 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 22:39:24.713608 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:39:24.713622 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:39:24.713704 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:39:24.713706 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:39:24.713798 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:39:24.713899 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:39:24.714149 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:39:24.714692 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:39:24.715619 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:39:24.715827 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:39:24.715904 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:39:24.716213 (+ 309us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:39:24.716832 (+ 619us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:39:24.716934 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 22:39:24.717007 (+ 73us) server_negotiation.cc:299] Negotiation successful
0925 22:39:24.717070 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260925 22:39:24.719779 23545 tablet.cc:2369] T bb84ba73ab9344b8b60560450f51c9b2 P f5beb7dcb6d641b1a4c9217cee5dd91b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:39:24.724984 23220 raft_consensus.cc:1261] T bb84ba73ab9344b8b60560450f51c9b2 P e240ef61505444dd98257c610c9e2702 [term 1 FOLLOWER]: Refusing update from remote peer eebff21e25664818b6cefcaa7d386fdc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:39:24.724989 23498 raft_consensus.cc:1261] T bb84ba73ab9344b8b60560450f51c9b2 P f5beb7dcb6d641b1a4c9217cee5dd91b [term 1 FOLLOWER]: Refusing update from remote peer eebff21e25664818b6cefcaa7d386fdc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:39:24.725723 23562 consensus_queue.cc:1035] T bb84ba73ab9344b8b60560450f51c9b2 P eebff21e25664818b6cefcaa7d386fdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "e240ef61505444dd98257c610c9e2702" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 41485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:39:24.725965 23565 consensus_queue.cc:1035] T bb84ba73ab9344b8b60560450f51c9b2 P eebff21e25664818b6cefcaa7d386fdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5beb7dcb6d641b1a4c9217cee5dd91b" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 41213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:39:24.733054 23567 mvcc.cc:205] Tried to move back new op lower bound from 7333379951503405056 to 7333379951396941824. Current Snapshot: MvccSnapshot[applied={T|T < 7333379951503405056}]
I20260925 22:39:24.733635 23568 mvcc.cc:205] Tried to move back new op lower bound from 7333379951503405056 to 7333379951396941824. Current Snapshot: MvccSnapshot[applied={T|T < 7333379951503405056}]
I20260925 22:39:24.736505 23569 mvcc.cc:205] Tried to move back new op lower bound from 7333379951503405056 to 7333379951396941824. Current Snapshot: MvccSnapshot[applied={T|T < 7333379951503405056}]
I20260925 22:39:24.743741 23082 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:39:24.743973 23082 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:39:24.746706 23082 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c836969096ff45db811a88fa616fd000: Sending DeleteTablet for 3 replicas of tablet bb84ba73ab9344b8b60560450f51c9b2
I20260925 22:39:24.747529 23346 tablet_service.cc:1514] Processing DeleteTablet for tablet bb84ba73ab9344b8b60560450f51c9b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:39:24 UTC) from {username='kudu', principal='kudu/127.6.246.254@KRBTEST.COM'} at 127.0.0.1:34154
I20260925 22:39:24.747515 23478 tablet_service.cc:1514] Processing DeleteTablet for tablet bb84ba73ab9344b8b60560450f51c9b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:39:24 UTC) from {username='kudu', principal='kudu/127.6.246.254@KRBTEST.COM'} at 127.0.0.1:38864
I20260925 22:39:24.747849 23207 tablet_service.cc:1514] Processing DeleteTablet for tablet bb84ba73ab9344b8b60560450f51c9b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:39:24 UTC) from {username='kudu', principal='kudu/127.6.246.254@KRBTEST.COM'} at 127.0.0.1:48860
I20260925 22:39:24.748342 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 23124
I20260925 22:39:24.749749 23577 tablet_replica.cc:330] T bb84ba73ab9344b8b60560450f51c9b2 P eebff21e25664818b6cefcaa7d386fdc: stopping tablet replica
I20260925 22:39:24.750196 23577 raft_consensus.cc:2229] T bb84ba73ab9344b8b60560450f51c9b2 P eebff21e25664818b6cefcaa7d386fdc [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:39:24.750659 23577 raft_consensus.cc:2258] T bb84ba73ab9344b8b60560450f51c9b2 P eebff21e25664818b6cefcaa7d386fdc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:39:24.752331 23577 ts_tablet_manager.cc:1905] T bb84ba73ab9344b8b60560450f51c9b2 P eebff21e25664818b6cefcaa7d386fdc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:39:24.755072 23577 ts_tablet_manager.cc:1918] T bb84ba73ab9344b8b60560450f51c9b2 P eebff21e25664818b6cefcaa7d386fdc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:39:24.755210 23577 log.cc:1196] T bb84ba73ab9344b8b60560450f51c9b2 P eebff21e25664818b6cefcaa7d386fdc: Deleting WAL directory at /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-1/wal/wals/bb84ba73ab9344b8b60560450f51c9b2
I20260925 22:39:24.755543 23577 ts_tablet_manager.cc:1939] T bb84ba73ab9344b8b60560450f51c9b2 P eebff21e25664818b6cefcaa7d386fdc: Deleting consensus metadata
I20260925 22:39:24.756608 23578 tablet_replica.cc:330] T bb84ba73ab9344b8b60560450f51c9b2 P f5beb7dcb6d641b1a4c9217cee5dd91b: stopping tablet replica
I20260925 22:39:24.757041 23070 catalog_manager.cc:4841] TS eebff21e25664818b6cefcaa7d386fdc (127.6.246.194:32827): tablet bb84ba73ab9344b8b60560450f51c9b2 (table test-table [id=61182d6ae946411db82fc4713fa1d394]) successfully deleted
I20260925 22:39:24.757145 23578 raft_consensus.cc:2229] T bb84ba73ab9344b8b60560450f51c9b2 P f5beb7dcb6d641b1a4c9217cee5dd91b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:39:24.757628 23578 raft_consensus.cc:2258] T bb84ba73ab9344b8b60560450f51c9b2 P f5beb7dcb6d641b1a4c9217cee5dd91b [term 1 FOLLOWER]: Raft consensus is shut down!
W20260925 22:39:24.759199 23072 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.246.193:41485 (error 108)
I20260925 22:39:24.759713 23578 ts_tablet_manager.cc:1905] T bb84ba73ab9344b8b60560450f51c9b2 P f5beb7dcb6d641b1a4c9217cee5dd91b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:39:24.759755 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 23278
I20260925 22:39:24.761633 23554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:24.761260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.193:41485 (local address 127.0.0.1:48864)
0925 22:39:24.761423 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 22:39:24.761506 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.193:41485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 22:39:24.761951 23072 catalog_manager.cc:4584] TS e240ef61505444dd98257c610c9e2702 (127.6.246.193:41485): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bb84ba73ab9344b8b60560450f51c9b2: Network error: Client connection negotiation failed: client connection to 127.6.246.193:41485: connect: Connection refused (error 111)
I20260925 22:39:24.765543 23578 ts_tablet_manager.cc:1918] T bb84ba73ab9344b8b60560450f51c9b2 P f5beb7dcb6d641b1a4c9217cee5dd91b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:39:24.765686 23578 log.cc:1196] T bb84ba73ab9344b8b60560450f51c9b2 P f5beb7dcb6d641b1a4c9217cee5dd91b: Deleting WAL directory at /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-2/wal/wals/bb84ba73ab9344b8b60560450f51c9b2
I20260925 22:39:24.766088 23578 ts_tablet_manager.cc:1939] T bb84ba73ab9344b8b60560450f51c9b2 P f5beb7dcb6d641b1a4c9217cee5dd91b: Deleting consensus metadata
I20260925 22:39:24.767957 23073 catalog_manager.cc:4841] TS f5beb7dcb6d641b1a4c9217cee5dd91b (127.6.246.195:41213): tablet bb84ba73ab9344b8b60560450f51c9b2 (table test-table [id=61182d6ae946411db82fc4713fa1d394]) successfully deleted
I20260925 22:39:24.768653 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 23415
I20260925 22:39:24.775522 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 23053
I20260925 22:39:44.783506 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:43767
--webserver_interface=127.6.246.254
--webserver_port=39397
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab
--principal=kudu/127.6.246.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:33735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.246.254:43767
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:39:44.901120 23581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:39:44.901454 23581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:39:44.901532 23581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:39:44.905709 23581 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 22:39:44.905790 23581 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 22:39:44.905820 23581 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 22:39:44.905857 23581 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:39:44.905901 23581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:39:44.905964 23581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:39:44.905994 23581 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:39:44.906038 23581 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:39:44.911927 23581 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:33735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.6.246.254:43767
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab
--principal=kudu/127.6.246.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:43767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.246.254
--webserver_port=39397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:39:44.913403 23581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:39:44.914687 23581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:39:44.925100 23589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:39:44.925060 23586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:39:44.925073 23587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:39:44.925079 23581 server_base.cc:1034] running on GCE node
I20260925 22:39:44.925976 23581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:39:44.927301 23581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:39:44.928519 23581 hybrid_clock.cc:648] HybridClock initialized: now 1790375984928487 us; error 38 us; skew 500 ppm
Sep 25 22:39:44 dist-test-slave-wjdr krb5kdc[22505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375984, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:39:44.939555 23581 init.cc:377] Logged in from keytab as kudu/127.6.246.254@KRBTEST.COM (short username kudu)
I20260925 22:39:44.940879 23581 webserver.cc:460] Webserver started at http://127.6.246.254:39397/ using document root <none> and password file <none>
I20260925 22:39:44.941588 23581 fs_manager.cc:359] Metadata directory not provided
I20260925 22:39:44.941680 23581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:39:44.946231 23581 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:39:44.948711 23596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:39:44.950181 23581 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:39:44.950349 23581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/master-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/master-0/wal
uuid: "c836969096ff45db811a88fa616fd000"
format_stamp: "Formatted at 2026-09-25 22:39:02 on dist-test-slave-wjdr"
server_key: "52344f895b6f9d8f0459de73d0663882"
server_key_iv: "970d6b85cc71460be704c5c6758a4c31"
server_key_version: "encryptionkey@0"
I20260925 22:39:44.951078 23581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:39:44.973155 23581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:39:44.976565 23581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:39:44.976811 23581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:39:44.985452 23581 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.254:43767
I20260925 22:39:44.985776 23648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.254:43767 every 8 connection(s)
I20260925 22:39:44.986793 23581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
I20260925 22:39:44.991526 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 23581
I20260925 22:39:44.992120 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.193:41485
--local_ip_for_outbound_sockets=127.6.246.193
--tserver_master_addrs=127.6.246.254:43767
--webserver_port=37015
--webserver_interface=127.6.246.193
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.193.keytab
--principal=kudu/127.6.246.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:33735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:39:44.992420 23649 sys_catalog.cc:263] Verifying existing consensus state
I20260925 22:39:44.995116 23649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c836969096ff45db811a88fa616fd000: Bootstrap starting.
I20260925 22:39:45.016316 23649 log.cc:824] T 00000000000000000000000000000000 P c836969096ff45db811a88fa616fd000: Log is configured to *not* fsync() on all Append() calls
I20260925 22:39:45.029250 23649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c836969096ff45db811a88fa616fd000: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 22:39:45.029736 23649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c836969096ff45db811a88fa616fd000: Bootstrap complete.
I20260925 22:39:45.034389 23649 raft_consensus.cc:348] T 00000000000000000000000000000000 P c836969096ff45db811a88fa616fd000 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c836969096ff45db811a88fa616fd000" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 43767 } }
I20260925 22:39:45.034861 23649 raft_consensus.cc:729] T 00000000000000000000000000000000 P c836969096ff45db811a88fa616fd000 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: c836969096ff45db811a88fa616fd000, State: Initialized, Role: FOLLOWER
I20260925 22:39:45.035429 23649 consensus_queue.cc:260] T 00000000000000000000000000000000 P c836969096ff45db811a88fa616fd000 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c836969096ff45db811a88fa616fd000" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 43767 } }
I20260925 22:39:45.035607 23649 raft_consensus.cc:388] T 00000000000000000000000000000000 P c836969096ff45db811a88fa616fd000 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:39:45.035692 23649 raft_consensus.cc:482] T 00000000000000000000000000000000 P c836969096ff45db811a88fa616fd000 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:39:45.035820 23649 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c836969096ff45db811a88fa616fd000 [term 2 FOLLOWER]: Advancing to term 3
I20260925 22:39:45.038038 23649 raft_consensus.cc:504] T 00000000000000000000000000000000 P c836969096ff45db811a88fa616fd000 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c836969096ff45db811a88fa616fd000" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 43767 } }
I20260925 22:39:45.038415 23649 leader_election.cc:302] T 00000000000000000000000000000000 P c836969096ff45db811a88fa616fd000 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c836969096ff45db811a88fa616fd000; no voters:
I20260925 22:39:45.038818 23649 leader_election.cc:288] T 00000000000000000000000000000000 P c836969096ff45db811a88fa616fd000 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 22:39:45.038921 23653 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c836969096ff45db811a88fa616fd000 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 22:39:45.039258 23653 raft_consensus.cc:686] T 00000000000000000000000000000000 P c836969096ff45db811a88fa616fd000 [term 3 LEADER]: Becoming Leader. State: Replica: c836969096ff45db811a88fa616fd000, State: Running, Role: LEADER
I20260925 22:39:45.039582 23653 consensus_queue.cc:237] T 00000000000000000000000000000000 P c836969096ff45db811a88fa616fd000 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c836969096ff45db811a88fa616fd000" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 43767 } }
I20260925 22:39:45.040251 23649 sys_catalog.cc:564] T 00000000000000000000000000000000 P c836969096ff45db811a88fa616fd000 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:39:45.041139 23654 sys_catalog.cc:455] T 00000000000000000000000000000000 P c836969096ff45db811a88fa616fd000 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "c836969096ff45db811a88fa616fd000" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c836969096ff45db811a88fa616fd000" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 43767 } } }
I20260925 22:39:45.041297 23654 sys_catalog.cc:458] T 00000000000000000000000000000000 P c836969096ff45db811a88fa616fd000 [sys.catalog]: This master's current role is: LEADER
I20260925 22:39:45.041692 23655 sys_catalog.cc:455] T 00000000000000000000000000000000 P c836969096ff45db811a88fa616fd000 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c836969096ff45db811a88fa616fd000. Latest consensus state: current_term: 3 leader_uuid: "c836969096ff45db811a88fa616fd000" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c836969096ff45db811a88fa616fd000" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 43767 } } }
I20260925 22:39:45.041792 23655 sys_catalog.cc:458] T 00000000000000000000000000000000 P c836969096ff45db811a88fa616fd000 [sys.catalog]: This master's current role is: LEADER
I20260925 22:39:45.042084 23662 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:39:45.048108 23662 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:39:45.048851 23662 catalog_manager.cc:1259] Loaded cluster ID: 1bf9af0db2514738bb75337fbf575332
I20260925 22:39:45.048928 23662 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:39:45.051702 23662 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:39:45.052506 23662 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c836969096ff45db811a88fa616fd000: Loaded TSK: 0
I20260925 22:39:45.052590 23662 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c836969096ff45db811a88fa616fd000: Loaded TSK: 1
F20260925 22:39:45.053473 23662 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f603366d9ad google::LogMessage::Fail() at ??:0
@ 0x7f603367088a google::LogMessage::SendToLog() at ??:0
@ 0x7f603366d4de google::LogMessage::Flush() at ??:0
@ 0x7f603366ea19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f603633118d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f6036329f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f603dfd27bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f603dfac99c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f603dfad178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f603dfeedf8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f603e00d82b std::function<>::operator()() at ??:0
@ 0x7f603dfacb4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f603dfadf3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f603dfaa579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f603dfee640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f60358bf3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f60358bfc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f60358c13f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f60358b0512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f6036ef46db start_thread at ??:0
@ 0x7f603216b61f clone at ??:0
W20260925 22:39:45.148967 23651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:39:45.149456 23651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:39:45.149540 23651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:39:45.156631 23651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:39:45.156718 23651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:39:45.156843 23651 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.193
I20260925 22:39:45.165946 23651 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:33735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.193.keytab
--principal=kudu/127.6.246.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.193:41485
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.246.193
--webserver_port=37015
--tserver_master_addrs=127.6.246.254:43767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23651
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.193
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:39:45.167776 23651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:39:45.169504 23651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:39:45.182353 23699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:39:45.182389 23696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:39:45.182399 23697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:39:45.182886 23651 server_base.cc:1034] running on GCE node
I20260925 22:39:45.183647 23651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:39:45.184616 23651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:39:45.185817 23651 hybrid_clock.cc:648] HybridClock initialized: now 1790375985185791 us; error 35 us; skew 500 ppm
Sep 25 22:39:45 dist-test-slave-wjdr krb5kdc[22505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375985, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:39:45.189831 23651 init.cc:377] Logged in from keytab as kudu/127.6.246.193@KRBTEST.COM (short username kudu)
I20260925 22:39:45.190935 23651 webserver.cc:460] Webserver started at http://127.6.246.193:37015/ using document root <none> and password file <none>
I20260925 22:39:45.191711 23651 fs_manager.cc:359] Metadata directory not provided
I20260925 22:39:45.191805 23651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:39:45.196830 23651 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:39:45.199519 23706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:39:45.201254 23651 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:39:45.201407 23651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-0/wal
uuid: "e240ef61505444dd98257c610c9e2702"
format_stamp: "Formatted at 2026-09-25 22:39:02 on dist-test-slave-wjdr"
server_key: "421a1f21ae2191645792feb592e8e95e"
server_key_iv: "40358199eb0614f96ca137c440a729a2"
server_key_version: "encryptionkey@0"
I20260925 22:39:45.202140 23651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:39:45.213871 23651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:39:45.217464 23651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:39:45.217692 23651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:39:45.218381 23651 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:39:45.219671 23713 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 22:39:45.226183 23651 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 22:39:45.226269 23651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.000s sys 0.000s
I20260925 22:39:45.226356 23651 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 22:39:45.228343 23651 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 22:39:45.228413 23651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260925 22:39:45.228915 23713 tablet_bootstrap.cc:492] T bb84ba73ab9344b8b60560450f51c9b2 P e240ef61505444dd98257c610c9e2702: Bootstrap starting.
I20260925 22:39:45.240959 23651 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.193:41485
I20260925 22:39:45.241250 23820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.193:41485 every 8 connection(s)
I20260925 22:39:45.242200 23651 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
I20260925 22:39:45.242798 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 23651
I20260925 22:39:45.243386 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.194:32827
--local_ip_for_outbound_sockets=127.6.246.194
--tserver_master_addrs=127.6.246.254:43767
--webserver_port=46653
--webserver_interface=127.6.246.194
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.194.keytab
--principal=kudu/127.6.246.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:33735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:39:45.254261 23713 log.cc:824] T bb84ba73ab9344b8b60560450f51c9b2 P e240ef61505444dd98257c610c9e2702: Log is configured to *not* fsync() on all Append() calls
I20260925 22:39:45.265069 23713 tablet_bootstrap.cc:492] T bb84ba73ab9344b8b60560450f51c9b2 P e240ef61505444dd98257c610c9e2702: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 22:39:45.265832 23713 tablet_bootstrap.cc:492] T bb84ba73ab9344b8b60560450f51c9b2 P e240ef61505444dd98257c610c9e2702: Bootstrap complete.
I20260925 22:39:45.267339 23713 ts_tablet_manager.cc:1397] T bb84ba73ab9344b8b60560450f51c9b2 P e240ef61505444dd98257c610c9e2702: Time spent bootstrapping tablet: real 0.039s user 0.026s sys 0.009s
I20260925 22:39:45.271095 23713 raft_consensus.cc:348] T bb84ba73ab9344b8b60560450f51c9b2 P e240ef61505444dd98257c610c9e2702 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5beb7dcb6d641b1a4c9217cee5dd91b" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 41213 } } peers { permanent_uuid: "eebff21e25664818b6cefcaa7d386fdc" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 32827 } } peers { permanent_uuid: "e240ef61505444dd98257c610c9e2702" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 41485 } }
I20260925 22:39:45.271721 23713 raft_consensus.cc:729] T bb84ba73ab9344b8b60560450f51c9b2 P e240ef61505444dd98257c610c9e2702 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e240ef61505444dd98257c610c9e2702, State: Initialized, Role: FOLLOWER
I20260925 22:39:45.272329 23713 consensus_queue.cc:260] T bb84ba73ab9344b8b60560450f51c9b2 P e240ef61505444dd98257c610c9e2702 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5beb7dcb6d641b1a4c9217cee5dd91b" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 41213 } } peers { permanent_uuid: "eebff21e25664818b6cefcaa7d386fdc" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 32827 } } peers { permanent_uuid: "e240ef61505444dd98257c610c9e2702" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 41485 } }
I20260925 22:39:45.273557 23713 ts_tablet_manager.cc:1428] T bb84ba73ab9344b8b60560450f51c9b2 P e240ef61505444dd98257c610c9e2702: Time spent starting tablet: real 0.006s user 0.005s sys 0.003s
W20260925 22:39:45.317904 23824 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:39:45.245692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43767 (local address 127.6.246.193:52735)
0925 22:39:45.247220 (+ 1528us) negotiation.cc:107] Waiting for socket to connect
0925 22:39:45.247288 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0925 22:39:45.248244 (+ 956us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:39:45.316910 (+ 68666us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43767: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":1223,"thread_start_us":182,"threads_started":1}
W20260925 22:39:45.318483 23821 heartbeater.cc:644] Failed to heartbeat to 127.6.246.254:43767 (0 consecutive failures): Network error: Failed to ping master at 127.6.246.254:43767: Client connection negotiation failed: client connection to 127.6.246.254:43767: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 22:39:45.319391 23824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:45.319088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43767 (local address 127.6.246.193:34979)
0925 22:39:45.319255 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 22:39:45.319329 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260925 22:39:45.320220 23824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:45.320018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43767 (local address 127.6.246.193:39413)
0925 22:39:45.320125 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0925 22:39:45.320171 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 22:39:45.320580 23821 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 22:39:45.376672 23823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:39:45.376981 23823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:39:45.377028 23823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:39:45.381472 23823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:39:45.381541 23823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:39:45.381631 23823 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.194
I20260925 22:39:45.387123 23823 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:33735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.194.keytab
--principal=kudu/127.6.246.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.194:32827
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.246.194
--webserver_port=46653
--tserver_master_addrs=127.6.246.254:43767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.194
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:39:45.388631 23823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:39:45.389942 23823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:39:45.401846 23832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:39:45.401829 23831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:39:45.401827 23834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:39:45.402304 23823 server_base.cc:1034] running on GCE node
I20260925 22:39:45.402815 23823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:39:45.403528 23823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:39:45.404687 23823 hybrid_clock.cc:648] HybridClock initialized: now 1790375985404668 us; error 35 us; skew 500 ppm
Sep 25 22:39:45 dist-test-slave-wjdr krb5kdc[22505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375985, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:39:45.408020 23823 init.cc:377] Logged in from keytab as kudu/127.6.246.194@KRBTEST.COM (short username kudu)
I20260925 22:39:45.409107 23823 webserver.cc:460] Webserver started at http://127.6.246.194:46653/ using document root <none> and password file <none>
I20260925 22:39:45.409803 23823 fs_manager.cc:359] Metadata directory not provided
I20260925 22:39:45.409893 23823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:39:45.414395 23823 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:39:45.416882 23841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:39:45.418390 23823 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:39:45.418557 23823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-1/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-1/wal
uuid: "eebff21e25664818b6cefcaa7d386fdc"
format_stamp: "Formatted at 2026-09-25 22:39:03 on dist-test-slave-wjdr"
server_key: "6c0e37dc988440bafb55579286a99c10"
server_key_iv: "436595a353af7510518876d9ff5a5084"
server_key_version: "encryptionkey@0"
I20260925 22:39:45.419224 23823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:39:45.444824 23823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:39:45.448900 23823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:39:45.449155 23823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:39:45.449929 23823 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:39:45.450940 23823 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:39:45.450997 23823 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:39:45.451043 23823 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:39:45.451094 23823 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:39:45.461504 23823 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.194:32827
I20260925 22:39:45.461539 23954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.194:32827 every 8 connection(s)
I20260925 22:39:45.462669 23823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
I20260925 22:39:45.463732 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 23823
I20260925 22:39:45.464265 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.195:41213
--local_ip_for_outbound_sockets=127.6.246.195
--tserver_master_addrs=127.6.246.254:43767
--webserver_port=43819
--webserver_interface=127.6.246.195
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.195.keytab
--principal=kudu/127.6.246.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:33735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:39:45.467973 23957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:45.465454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43767 (local address 127.6.246.194:58931)
0925 22:39:45.466545 (+ 1091us) negotiation.cc:107] Waiting for socket to connect
0925 22:39:45.466730 (+ 185us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":759,"thread_start_us":159,"threads_started":1}
W20260925 22:39:45.468597 23955 heartbeater.cc:644] Failed to heartbeat to 127.6.246.254:43767 (0 consecutive failures): Network error: Failed to ping master at 127.6.246.254:43767: Client connection negotiation failed: client connection to 127.6.246.254:43767: connect: Connection refused (error 111)
I20260925 22:39:45.469455 23957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:45.469100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43767 (local address 127.6.246.194:54681)
0925 22:39:45.469269 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 22:39:45.469352 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260925 22:39:45.470324 23957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:45.469983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43767 (local address 127.6.246.194:35711)
0925 22:39:45.470143 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 22:39:45.470220 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260925 22:39:45.470517 23955 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 22:39:45.596045 23958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:39:45.596357 23958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:39:45.596405 23958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:39:45.601778 23958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:39:45.601893 23958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:39:45.602020 23958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.195
I20260925 22:39:45.608007 23958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:33735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.195.keytab
--principal=kudu/127.6.246.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.195:41213
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.246.195
--webserver_port=43819
--tserver_master_addrs=127.6.246.254:43767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.195
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:39:45.609414 23958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:39:45.610656 23958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:39:45.621853 23966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:39:45.621861 23964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:39:45.621827 23963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:39:45.622053 23958 server_base.cc:1034] running on GCE node
I20260925 22:39:45.622545 23958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:39:45.623365 23958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:39:45.624533 23958 hybrid_clock.cc:648] HybridClock initialized: now 1790375985624497 us; error 46 us; skew 500 ppm
Sep 25 22:39:45 dist-test-slave-wjdr krb5kdc[22505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375985, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:39:45.627540 23958 init.cc:377] Logged in from keytab as kudu/127.6.246.195@KRBTEST.COM (short username kudu)
I20260925 22:39:45.628593 23958 webserver.cc:460] Webserver started at http://127.6.246.195:43819/ using document root <none> and password file <none>
I20260925 22:39:45.629206 23958 fs_manager.cc:359] Metadata directory not provided
I20260925 22:39:45.629266 23958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:39:45.633477 23958 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260925 22:39:45.635650 23973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:39:45.637113 23958 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:39:45.637247 23958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-2/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-2/wal
uuid: "f5beb7dcb6d641b1a4c9217cee5dd91b"
format_stamp: "Formatted at 2026-09-25 22:39:03 on dist-test-slave-wjdr"
server_key: "07a8b4a7b52202ddefcb5a6c904bac9d"
server_key_iv: "92ac20f6012d9991b06ab55746be1658"
server_key_version: "encryptionkey@0"
I20260925 22:39:45.637836 23958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:39:45.660179 23958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:39:45.663658 23958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:39:45.663887 23958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:39:45.664623 23958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:39:45.665604 23958 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:39:45.665653 23958 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:39:45.665730 23958 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:39:45.665772 23958 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:39:45.676332 23958 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.195:41213
I20260925 22:39:45.676363 24086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.195:41213 every 8 connection(s)
I20260925 22:39:45.677520 23958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
I20260925 22:39:45.681720 24089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:45.679977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43767 (local address 127.6.246.195:44777)
0925 22:39:45.680794 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0925 22:39:45.680919 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":125,"threads_started":1}
W20260925 22:39:45.682265 24087 heartbeater.cc:644] Failed to heartbeat to 127.6.246.254:43767 (0 consecutive failures): Network error: Failed to ping master at 127.6.246.254:43767: Client connection negotiation failed: client connection to 127.6.246.254:43767: connect: Connection refused (error 111)
I20260925 22:39:45.682895 24089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:45.682670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43767 (local address 127.6.246.195:56341)
0925 22:39:45.682783 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0925 22:39:45.682835 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 22:39:45.683569 24089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:45.683348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43767 (local address 127.6.246.195:60809)
0925 22:39:45.683453 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0925 22:39:45.683519 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 22:39:45.683578 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 23958
W20260925 22:39:45.683805 24087 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 22:39:46.321925 24091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:46.321359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43767 (local address 127.6.246.193:58843)
0925 22:39:46.321697 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0925 22:39:46.321772 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":99,"threads_started":1}
I20260925 22:39:46.472016 24092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:46.471440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43767 (local address 127.6.246.194:42611)
0925 22:39:46.471790 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0925 22:39:46.471867 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":107,"threads_started":1}
I20260925 22:39:46.685138 24093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:46.684614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43767 (local address 127.6.246.195:49387)
0925 22:39:46.684925 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0925 22:39:46.685005 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":83,"threads_started":1}
I20260925 22:39:47.029484 24094 raft_consensus.cc:482] T bb84ba73ab9344b8b60560450f51c9b2 P e240ef61505444dd98257c610c9e2702 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:39:47.029742 24094 raft_consensus.cc:504] T bb84ba73ab9344b8b60560450f51c9b2 P e240ef61505444dd98257c610c9e2702 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5beb7dcb6d641b1a4c9217cee5dd91b" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 41213 } } peers { permanent_uuid: "eebff21e25664818b6cefcaa7d386fdc" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 32827 } } peers { permanent_uuid: "e240ef61505444dd98257c610c9e2702" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 41485 } }
I20260925 22:39:47.031111 24094 leader_election.cc:288] T bb84ba73ab9344b8b60560450f51c9b2 P e240ef61505444dd98257c610c9e2702 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f5beb7dcb6d641b1a4c9217cee5dd91b (127.6.246.195:41213), eebff21e25664818b6cefcaa7d386fdc (127.6.246.194:32827)
Sep 25 22:39:47 dist-test-slave-wjdr krb5kdc[22505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375985, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.193@KRBTEST.COM for kudu/127.6.246.194@KRBTEST.COM
Sep 25 22:39:47 dist-test-slave-wjdr krb5kdc[22505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790375985, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.193@KRBTEST.COM for kudu/127.6.246.195@KRBTEST.COM
I20260925 22:39:47.046235 24095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:47.031436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.195:41213 (local address 127.6.246.193:42075)
0925 22:39:47.031813 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0925 22:39:47.031830 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:39:47.031970 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:39:47.034019 (+ 2049us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:39:47.034026 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:39:47.034611 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:39:47.035124 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:39:47.035140 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:39:47.036522 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:39:47.036525 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:39:47.037555 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:39:47.037565 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:39:47.038291 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:39:47.038295 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:39:47.038531 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:39:47.039285 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:39:47.039302 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:39:47.041464 (+ 2162us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:39:47.043808 (+ 2344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:39:47.043812 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:39:47.043814 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:39:47.044013 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:39:47.044420 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:39:47.044423 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:39:47.044424 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:39:47.044526 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:39:47.045267 (+ 741us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:39:47.045273 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:39:47.045500 (+ 227us) client_negotiation.cc:769] Sending connection context
0925 22:39:47.045695 (+ 195us) client_negotiation.cc:240] Negotiation successful
0925 22:39:47.046022 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"mutex_wait_us":197,"spinlock_wait_cycles":29312,"thread_start_us":90,"threads_started":1}
I20260925 22:39:47.046285 24098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:47.031837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.193:42075 (local address 127.6.246.195:41213)
0925 22:39:47.032247 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0925 22:39:47.032268 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 22:39:47.032302 (+ 34us) server_negotiation.cc:407] Connection header received
0925 22:39:47.033069 (+ 767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:39:47.033077 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:39:47.033545 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:39:47.034508 (+ 963us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:39:47.035308 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:39:47.036339 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:39:47.037691 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:39:47.038156 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:39:47.038347 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:39:47.041651 (+ 3304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:39:47.041656 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:39:47.041666 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:39:47.041695 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:39:47.043645 (+ 1950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:39:47.044152 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:39:47.044157 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:39:47.044161 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:39:47.044306 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:39:47.044665 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:39:47.044668 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:39:47.044670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:39:47.045146 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:39:47.045290 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 22:39:47.045821 (+ 531us) server_negotiation.cc:299] Negotiation successful
0925 22:39:47.046095 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":100,"threads_started":1}
I20260925 22:39:47.046989 24096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:47.031437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.194:32827 (local address 127.6.246.193:53991)
0925 22:39:47.031903 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0925 22:39:47.031916 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:39:47.031998 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:39:47.034038 (+ 2040us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:39:47.034042 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:39:47.034604 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:39:47.035123 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:39:47.035139 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:39:47.036512 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:39:47.036515 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:39:47.037555 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:39:47.037564 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:39:47.038291 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:39:47.038295 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:39:47.038533 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:39:47.039338 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:39:47.039350 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:39:47.041015 (+ 1665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:39:47.043523 (+ 2508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:39:47.043529 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:39:47.043543 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:39:47.043887 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:39:47.044295 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:39:47.044298 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:39:47.044299 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:39:47.044494 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:39:47.046421 (+ 1927us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:39:47.046426 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:39:47.046529 (+ 103us) client_negotiation.cc:769] Sending connection context
0925 22:39:47.046666 (+ 137us) client_negotiation.cc:240] Negotiation successful
0925 22:39:47.046815 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"mutex_wait_us":270,"spinlock_wait_cycles":4352,"thread_start_us":92,"threads_started":1}
I20260925 22:39:47.047322 24097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:47.031612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.193:53991 (local address 127.6.246.194:32827)
0925 22:39:47.032122 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0925 22:39:47.032139 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 22:39:47.032185 (+ 46us) server_negotiation.cc:407] Connection header received
0925 22:39:47.033070 (+ 885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:39:47.033077 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:39:47.033547 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:39:47.033923 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:39:47.035321 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:39:47.036336 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:39:47.037680 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:39:47.038157 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:39:47.038347 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:39:47.041287 (+ 2940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:39:47.041293 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:39:47.041305 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:39:47.041334 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:39:47.043382 (+ 2048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:39:47.044013 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:39:47.044019 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:39:47.044024 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:39:47.044188 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:39:47.044580 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:39:47.044584 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:39:47.044585 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:39:47.045147 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:39:47.045282 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 22:39:47.046812 (+ 1530us) server_negotiation.cc:299] Negotiation successful
0925 22:39:47.047104 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":90,"threads_started":1}
W20260925 22:39:47.049463 23707 leader_election.cc:341] T bb84ba73ab9344b8b60560450f51c9b2 P e240ef61505444dd98257c610c9e2702 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eebff21e25664818b6cefcaa7d386fdc (127.6.246.194:32827): Not found: Tablet not found: bb84ba73ab9344b8b60560450f51c9b2
W20260925 22:39:47.049687 23710 leader_election.cc:341] T bb84ba73ab9344b8b60560450f51c9b2 P e240ef61505444dd98257c610c9e2702 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f5beb7dcb6d641b1a4c9217cee5dd91b (127.6.246.195:41213): Not found: Tablet not found: bb84ba73ab9344b8b60560450f51c9b2
I20260925 22:39:47.049737 23710 leader_election.cc:302] T bb84ba73ab9344b8b60560450f51c9b2 P e240ef61505444dd98257c610c9e2702 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e240ef61505444dd98257c610c9e2702; no voters: eebff21e25664818b6cefcaa7d386fdc, f5beb7dcb6d641b1a4c9217cee5dd91b
I20260925 22:39:47.049993 24094 raft_consensus.cc:2726] T bb84ba73ab9344b8b60560450f51c9b2 P e240ef61505444dd98257c610c9e2702 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 22:39:47.323274 24096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:47.322913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43767 (local address 127.6.246.193:59787)
0925 22:39:47.323087 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 22:39:47.323163 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260925 22:39:47.473668 24099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:47.473099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43767 (local address 127.6.246.194:33981)
0925 22:39:47.473428 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0925 22:39:47.473504 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":102,"threads_started":1}
I20260925 22:39:47.686503 24100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:47.685946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43767 (local address 127.6.246.195:49745)
0925 22:39:47.686290 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0925 22:39:47.686352 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":106,"threads_started":1}
I20260925 22:39:48.324617 24101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:48.324090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43767 (local address 127.6.246.193:46241)
0925 22:39:48.324376 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0925 22:39:48.324434 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":100,"threads_started":1}
I20260925 22:39:48.475174 24102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:48.474616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43767 (local address 127.6.246.194:37973)
0925 22:39:48.474960 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0925 22:39:48.475037 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":95,"threads_started":1}
I20260925 22:39:48.688022 24103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:48.687457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43767 (local address 127.6.246.195:46083)
0925 22:39:48.687772 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0925 22:39:48.687850 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":117,"threads_started":1}
I20260925 22:39:48.704044 24104 raft_consensus.cc:482] T bb84ba73ab9344b8b60560450f51c9b2 P e240ef61505444dd98257c610c9e2702 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:39:48.704236 24104 raft_consensus.cc:504] T bb84ba73ab9344b8b60560450f51c9b2 P e240ef61505444dd98257c610c9e2702 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5beb7dcb6d641b1a4c9217cee5dd91b" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 41213 } } peers { permanent_uuid: "eebff21e25664818b6cefcaa7d386fdc" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 32827 } } peers { permanent_uuid: "e240ef61505444dd98257c610c9e2702" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 41485 } }
I20260925 22:39:48.704793 24104 leader_election.cc:288] T bb84ba73ab9344b8b60560450f51c9b2 P e240ef61505444dd98257c610c9e2702 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f5beb7dcb6d641b1a4c9217cee5dd91b (127.6.246.195:41213), eebff21e25664818b6cefcaa7d386fdc (127.6.246.194:32827)
W20260925 22:39:48.705808 23710 leader_election.cc:341] T bb84ba73ab9344b8b60560450f51c9b2 P e240ef61505444dd98257c610c9e2702 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f5beb7dcb6d641b1a4c9217cee5dd91b (127.6.246.195:41213): Not found: Tablet not found: bb84ba73ab9344b8b60560450f51c9b2
W20260925 22:39:48.705969 23707 leader_election.cc:341] T bb84ba73ab9344b8b60560450f51c9b2 P e240ef61505444dd98257c610c9e2702 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eebff21e25664818b6cefcaa7d386fdc (127.6.246.194:32827): Not found: Tablet not found: bb84ba73ab9344b8b60560450f51c9b2
I20260925 22:39:48.706174 23707 leader_election.cc:302] T bb84ba73ab9344b8b60560450f51c9b2 P e240ef61505444dd98257c610c9e2702 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e240ef61505444dd98257c610c9e2702; no voters: eebff21e25664818b6cefcaa7d386fdc, f5beb7dcb6d641b1a4c9217cee5dd91b
I20260925 22:39:48.706333 24104 raft_consensus.cc:2726] T bb84ba73ab9344b8b60560450f51c9b2 P e240ef61505444dd98257c610c9e2702 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 22:39:49.326242 24105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:49.325659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43767 (local address 127.6.246.193:53543)
0925 22:39:49.325993 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0925 22:39:49.326057 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":128,"threads_started":1}
I20260925 22:39:49.476675 24106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:49.476162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43767 (local address 127.6.246.194:43113)
0925 22:39:49.476450 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0925 22:39:49.476532 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":95,"threads_started":1}
I20260925 22:39:49.689462 24107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:49.688892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43767 (local address 127.6.246.195:55403)
0925 22:39:49.689212 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0925 22:39:49.689274 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":113,"threads_started":1}
I20260925 22:39:50.264635 24108 raft_consensus.cc:482] T bb84ba73ab9344b8b60560450f51c9b2 P e240ef61505444dd98257c610c9e2702 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:39:50.264813 24108 raft_consensus.cc:504] T bb84ba73ab9344b8b60560450f51c9b2 P e240ef61505444dd98257c610c9e2702 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5beb7dcb6d641b1a4c9217cee5dd91b" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 41213 } } peers { permanent_uuid: "eebff21e25664818b6cefcaa7d386fdc" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 32827 } } peers { permanent_uuid: "e240ef61505444dd98257c610c9e2702" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 41485 } }
I20260925 22:39:50.265426 24108 leader_election.cc:288] T bb84ba73ab9344b8b60560450f51c9b2 P e240ef61505444dd98257c610c9e2702 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f5beb7dcb6d641b1a4c9217cee5dd91b (127.6.246.195:41213), eebff21e25664818b6cefcaa7d386fdc (127.6.246.194:32827)
W20260925 22:39:50.266460 23707 leader_election.cc:341] T bb84ba73ab9344b8b60560450f51c9b2 P e240ef61505444dd98257c610c9e2702 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eebff21e25664818b6cefcaa7d386fdc (127.6.246.194:32827): Not found: Tablet not found: bb84ba73ab9344b8b60560450f51c9b2
W20260925 22:39:50.266650 23710 leader_election.cc:341] T bb84ba73ab9344b8b60560450f51c9b2 P e240ef61505444dd98257c610c9e2702 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f5beb7dcb6d641b1a4c9217cee5dd91b (127.6.246.195:41213): Not found: Tablet not found: bb84ba73ab9344b8b60560450f51c9b2
I20260925 22:39:50.266731 23710 leader_election.cc:302] T bb84ba73ab9344b8b60560450f51c9b2 P e240ef61505444dd98257c610c9e2702 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e240ef61505444dd98257c610c9e2702; no voters: eebff21e25664818b6cefcaa7d386fdc, f5beb7dcb6d641b1a4c9217cee5dd91b
I20260925 22:39:50.266906 24108 raft_consensus.cc:2726] T bb84ba73ab9344b8b60560450f51c9b2 P e240ef61505444dd98257c610c9e2702 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 22:39:50.328074 24109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:50.327341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43767 (local address 127.6.246.193:52599)
0925 22:39:50.327864 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0925 22:39:50.327953 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":136,"threads_started":1}
I20260925 22:39:50.478246 24110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:50.477608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43767 (local address 127.6.246.194:58405)
0925 22:39:50.478028 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0925 22:39:50.478123 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":101,"threads_started":1}
I20260925 22:39:50.691051 24111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:50.690463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43767 (local address 127.6.246.195:53715)
0925 22:39:50.690777 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0925 22:39:50.690874 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":120,"threads_started":1}
I20260925 22:39:51.329803 24112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:51.329188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43767 (local address 127.6.246.193:52403)
0925 22:39:51.329572 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0925 22:39:51.329663 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":95,"threads_started":1}
I20260925 22:39:51.480044 24113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:51.479369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43767 (local address 127.6.246.194:51311)
0925 22:39:51.479794 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0925 22:39:51.479917 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":102,"threads_started":1}
I20260925 22:39:51.692976 24114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:51.692247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43767 (local address 127.6.246.195:40497)
0925 22:39:51.692697 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0925 22:39:51.692846 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":127,"threads_started":1}
I20260925 22:39:52.331259 24115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:52.330791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43767 (local address 127.6.246.193:53983)
0925 22:39:52.331071 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0925 22:39:52.331127 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":98,"threads_started":1}
I20260925 22:39:52.481762 24116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:52.481140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43767 (local address 127.6.246.194:46905)
0925 22:39:52.481534 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0925 22:39:52.481596 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":141,"threads_started":1}
I20260925 22:39:52.694485 24117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:52.693862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43767 (local address 127.6.246.195:52017)
0925 22:39:52.694256 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0925 22:39:52.694319 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":124,"threads_started":1}
I20260925 22:39:52.775914 24118 raft_consensus.cc:482] T bb84ba73ab9344b8b60560450f51c9b2 P e240ef61505444dd98257c610c9e2702 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:39:52.776134 24118 raft_consensus.cc:504] T bb84ba73ab9344b8b60560450f51c9b2 P e240ef61505444dd98257c610c9e2702 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5beb7dcb6d641b1a4c9217cee5dd91b" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 41213 } } peers { permanent_uuid: "eebff21e25664818b6cefcaa7d386fdc" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 32827 } } peers { permanent_uuid: "e240ef61505444dd98257c610c9e2702" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 41485 } }
I20260925 22:39:52.776767 24118 leader_election.cc:288] T bb84ba73ab9344b8b60560450f51c9b2 P e240ef61505444dd98257c610c9e2702 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f5beb7dcb6d641b1a4c9217cee5dd91b (127.6.246.195:41213), eebff21e25664818b6cefcaa7d386fdc (127.6.246.194:32827)
W20260925 22:39:52.777675 23710 leader_election.cc:341] T bb84ba73ab9344b8b60560450f51c9b2 P e240ef61505444dd98257c610c9e2702 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f5beb7dcb6d641b1a4c9217cee5dd91b (127.6.246.195:41213): Not found: Tablet not found: bb84ba73ab9344b8b60560450f51c9b2
W20260925 22:39:52.777813 23707 leader_election.cc:341] T bb84ba73ab9344b8b60560450f51c9b2 P e240ef61505444dd98257c610c9e2702 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eebff21e25664818b6cefcaa7d386fdc (127.6.246.194:32827): Not found: Tablet not found: bb84ba73ab9344b8b60560450f51c9b2
I20260925 22:39:52.777915 23707 leader_election.cc:302] T bb84ba73ab9344b8b60560450f51c9b2 P e240ef61505444dd98257c610c9e2702 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e240ef61505444dd98257c610c9e2702; no voters: eebff21e25664818b6cefcaa7d386fdc, f5beb7dcb6d641b1a4c9217cee5dd91b
I20260925 22:39:52.778105 24118 raft_consensus.cc:2726] T bb84ba73ab9344b8b60560450f51c9b2 P e240ef61505444dd98257c610c9e2702 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 22:39:53.332990 24119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:53.332395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43767 (local address 127.6.246.193:48321)
0925 22:39:53.332764 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0925 22:39:53.332824 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":185,"threads_started":1}
I20260925 22:39:53.483259 24120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:53.482719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43767 (local address 127.6.246.194:59211)
0925 22:39:53.483046 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0925 22:39:53.483127 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":122,"threads_started":1}
I20260925 22:39:53.695860 24121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:53.695387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43767 (local address 127.6.246.195:60823)
0925 22:39:53.695674 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0925 22:39:53.695733 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":95,"threads_started":1}
I20260925 22:39:54.334374 24122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:54.333880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43767 (local address 127.6.246.193:47733)
0925 22:39:54.334194 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0925 22:39:54.334255 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":102,"threads_started":1}
I20260925 22:39:54.484725 24123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:54.484142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43767 (local address 127.6.246.194:36801)
0925 22:39:54.484549 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0925 22:39:54.484615 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":121,"threads_started":1}
I20260925 22:39:54.697153 24124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:54.696704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43767 (local address 127.6.246.195:59301)
0925 22:39:54.696991 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0925 22:39:54.697046 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":106,"threads_started":1}
I20260925 22:39:55.336014 24125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:55.335430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43767 (local address 127.6.246.193:56161)
0925 22:39:55.335806 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0925 22:39:55.335899 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":102,"threads_started":1}
I20260925 22:39:55.486349 24126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:55.485714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43767 (local address 127.6.246.194:51099)
0925 22:39:55.486135 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0925 22:39:55.486225 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":107,"threads_started":1}
I20260925 22:39:55.698596 24127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:55.698085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43767 (local address 127.6.246.195:54471)
0925 22:39:55.698390 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0925 22:39:55.698469 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":83,"threads_started":1}
I20260925 22:39:56.337464 24128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:56.336979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43767 (local address 127.6.246.193:49203)
0925 22:39:56.337280 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0925 22:39:56.337340 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":101,"threads_started":1}
I20260925 22:39:56.487773 24129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:56.487241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43767 (local address 127.6.246.194:46379)
0925 22:39:56.487546 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0925 22:39:56.487611 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":114,"threads_started":1}
I20260925 22:39:56.699990 24130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:56.699519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43767 (local address 127.6.246.195:48173)
0925 22:39:56.699809 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0925 22:39:56.699868 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":87,"threads_started":1}
I20260925 22:39:57.339025 24131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:57.338497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43767 (local address 127.6.246.193:42735)
0925 22:39:57.338845 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0925 22:39:57.338902 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":124,"threads_started":1}
I20260925 22:39:57.489326 24132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:57.488773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43767 (local address 127.6.246.194:37699)
0925 22:39:57.489120 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0925 22:39:57.489191 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":119,"threads_started":1}
I20260925 22:39:57.701543 24133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:57.701016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43767 (local address 127.6.246.195:35223)
0925 22:39:57.701341 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0925 22:39:57.701415 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":99,"threads_started":1}
I20260925 22:39:58.340848 24134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:58.340187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43767 (local address 127.6.246.193:47115)
0925 22:39:58.340615 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0925 22:39:58.340709 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":98,"threads_started":1}
I20260925 22:39:58.490778 24135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:58.490261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43767 (local address 127.6.246.194:55547)
0925 22:39:58.490573 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0925 22:39:58.490644 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":96,"threads_started":1}
I20260925 22:39:58.549328 24136 raft_consensus.cc:482] T bb84ba73ab9344b8b60560450f51c9b2 P e240ef61505444dd98257c610c9e2702 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:39:58.549506 24136 raft_consensus.cc:504] T bb84ba73ab9344b8b60560450f51c9b2 P e240ef61505444dd98257c610c9e2702 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5beb7dcb6d641b1a4c9217cee5dd91b" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 41213 } } peers { permanent_uuid: "eebff21e25664818b6cefcaa7d386fdc" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 32827 } } peers { permanent_uuid: "e240ef61505444dd98257c610c9e2702" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 41485 } }
I20260925 22:39:58.550078 24136 leader_election.cc:288] T bb84ba73ab9344b8b60560450f51c9b2 P e240ef61505444dd98257c610c9e2702 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f5beb7dcb6d641b1a4c9217cee5dd91b (127.6.246.195:41213), eebff21e25664818b6cefcaa7d386fdc (127.6.246.194:32827)
W20260925 22:39:58.551023 23707 leader_election.cc:341] T bb84ba73ab9344b8b60560450f51c9b2 P e240ef61505444dd98257c610c9e2702 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eebff21e25664818b6cefcaa7d386fdc (127.6.246.194:32827): Not found: Tablet not found: bb84ba73ab9344b8b60560450f51c9b2
W20260925 22:39:58.551129 23710 leader_election.cc:341] T bb84ba73ab9344b8b60560450f51c9b2 P e240ef61505444dd98257c610c9e2702 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f5beb7dcb6d641b1a4c9217cee5dd91b (127.6.246.195:41213): Not found: Tablet not found: bb84ba73ab9344b8b60560450f51c9b2
I20260925 22:39:58.551236 23710 leader_election.cc:302] T bb84ba73ab9344b8b60560450f51c9b2 P e240ef61505444dd98257c610c9e2702 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e240ef61505444dd98257c610c9e2702; no voters: eebff21e25664818b6cefcaa7d386fdc, f5beb7dcb6d641b1a4c9217cee5dd91b
I20260925 22:39:58.551409 24136 raft_consensus.cc:2726] T bb84ba73ab9344b8b60560450f51c9b2 P e240ef61505444dd98257c610c9e2702 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 22:39:58.702927 24137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:58.702426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43767 (local address 127.6.246.195:46409)
0925 22:39:58.702745 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0925 22:39:58.702804 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":136,"threads_started":1}
I20260925 22:39:59.342433 24138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:59.341834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43767 (local address 127.6.246.193:44727)
0925 22:39:59.342234 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0925 22:39:59.342313 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":144,"threads_started":1}
I20260925 22:39:59.492313 24139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:59.491721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43767 (local address 127.6.246.194:58491)
0925 22:39:59.492104 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0925 22:39:59.492176 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":134,"threads_started":1}
I20260925 22:39:59.704396 24140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:39:59.703873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43767 (local address 127.6.246.195:47521)
0925 22:39:59.704209 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0925 22:39:59.704286 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":104,"threads_started":1}
I20260925 22:40:00.344193 24141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:00.343679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43767 (local address 127.6.246.193:40493)
0925 22:40:00.344010 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:00.344085 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":96,"threads_started":1}
I20260925 22:40:00.493988 24142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:00.493386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:43767 (local address 127.6.246.194:52525)
0925 22:40:00.493760 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:00.493862 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.254:43767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":118,"threads_started":1}
I20260925 22:40:00.688798 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 23651
I20260925 22:40:00.695830 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 23823
I20260925 22:40:00.701699 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 23958
2026-09-25T22:40:00Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59654 ms)
[----------] 30 tests from SecurityITest (247900 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:40:00 dist-test-slave-wjdr krb5kdc[24145](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:40:00 dist-test-slave-wjdr krb5kdc[24145](info): set up 2 sockets
Sep 25 22:40:00 dist-test-slave-wjdr krb5kdc[24145](info): commencing operation
krb5kdc: starting...
W20260925 22:40:01.748674 7131 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Sep 25 22:40:01 dist-test-slave-wjdr krb5kdc[24145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:40:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:40:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.246.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab.
Entry for principal kudu/127.6.246.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab.
I20260925 22:40:01.884225 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:46309
--webserver_interface=127.6.246.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab
--principal=kudu/127.6.246.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:44821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.246.254:46309
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:40:02.021220 24158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:40:02.021749 24158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:40:02.021842 24158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:40:02.027302 24158 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:40:02.027432 24158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:40:02.027475 24158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:40:02.027513 24158 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:40:02.027575 24158 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:40:02.034140 24158 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:44821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.246.254:46309
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab
--principal=kudu/127.6.246.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:46309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.246.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:40:02.035672 24158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:40:02.037101 24158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:40:02.048873 24163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:40:02.048946 24158 server_base.cc:1034] running on GCE node
W20260925 22:40:02.049248 24164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:02.049341 24166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:40:02.050199 24158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:40:02.052011 24158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:40:02.053154 24158 hybrid_clock.cc:648] HybridClock initialized: now 1790376002053125 us; error 55 us; skew 500 ppm
Sep 25 22:40:02 dist-test-slave-wjdr krb5kdc[24145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376002, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:40:02.057632 24158 init.cc:377] Logged in from keytab as kudu/127.6.246.254@KRBTEST.COM (short username kudu)
I20260925 22:40:02.059190 24158 webserver.cc:460] Webserver started at http://127.6.246.254:45855/ using document root <none> and password file <none>
I20260925 22:40:02.059998 24158 fs_manager.cc:359] Metadata directory not provided
I20260925 22:40:02.060086 24158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:40:02.060384 24158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:40:02.063274 24158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/minicluster-data/master-0/data/instance:
uuid: "5e7606b909d1437fba2b13c1c9d896d2"
format_stamp: "Formatted at 2026-09-25 22:40:02 on dist-test-slave-wjdr"
server_key: "068e6efdb34dba2f3d6028fc5d95196a"
server_key_iv: "d0d419667102d2d84cce6c4938e4bb39"
server_key_version: "encryptionkey@0"
I20260925 22:40:02.064013 24158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/minicluster-data/master-0/wal/instance:
uuid: "5e7606b909d1437fba2b13c1c9d896d2"
format_stamp: "Formatted at 2026-09-25 22:40:02 on dist-test-slave-wjdr"
server_key: "068e6efdb34dba2f3d6028fc5d95196a"
server_key_iv: "d0d419667102d2d84cce6c4938e4bb39"
server_key_version: "encryptionkey@0"
I20260925 22:40:02.068826 24158 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260925 22:40:02.071738 24173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:02.073411 24158 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:40:02.073570 24158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/minicluster-data/master-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/minicluster-data/master-0/wal
uuid: "5e7606b909d1437fba2b13c1c9d896d2"
format_stamp: "Formatted at 2026-09-25 22:40:02 on dist-test-slave-wjdr"
server_key: "068e6efdb34dba2f3d6028fc5d95196a"
server_key_iv: "d0d419667102d2d84cce6c4938e4bb39"
server_key_version: "encryptionkey@0"
I20260925 22:40:02.073697 24158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:40:02.103715 24158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:40:02.107784 24158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:40:02.108060 24158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:40:02.117148 24158 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.254:46309
I20260925 22:40:02.117156 24225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.254:46309 every 8 connection(s)
I20260925 22:40:02.118386 24158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
I20260925 22:40:02.122051 24226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:40:02.123358 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 24158
I20260925 22:40:02.123546 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/minicluster-data/master-0/wal/instance
I20260925 22:40:02.123926 7131 external_mini_cluster.cc:1428] Setting key 2ca444d799679005174a02d677bf3340
I20260925 22:40:02.129854 24226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e7606b909d1437fba2b13c1c9d896d2: Bootstrap starting.
I20260925 22:40:02.133226 24226 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e7606b909d1437fba2b13c1c9d896d2: Neither blocks nor log segments found. Creating new log.
Sep 25 22:40:02 dist-test-slave-wjdr krb5kdc[24145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
I20260925 22:40:02.134265 24226 log.cc:824] T 00000000000000000000000000000000 P 5e7606b909d1437fba2b13c1c9d896d2: Log is configured to *not* fsync() on all Append() calls
I20260925 22:40:02.136438 24226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e7606b909d1437fba2b13c1c9d896d2: No bootstrap required, opened a new log
I20260925 22:40:02.139880 24229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:02.125678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38108 (local address 127.6.246.254:46309)
0925 22:40:02.126234 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:02.126257 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:02.126309 (+ 52us) server_negotiation.cc:407] Connection header received
0925 22:40:02.127028 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:02.127046 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:02.127422 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:02.127736 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:02.128769 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:02.129847 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:02.131067 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:02.131562 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:02.131693 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:02.134899 (+ 3206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:02.134905 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:02.134921 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:02.134949 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:02.136855 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:02.137309 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:02.137314 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:02.137319 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:02.137430 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:02.137723 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:02.137727 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:02.137729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:02.138154 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:02.138378 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:02.138616 (+ 238us) server_negotiation.cc:299] Negotiation successful
0925 22:40:02.138936 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":181,"threads_started":1}
I20260925 22:40:02.140667 24226 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e7606b909d1437fba2b13c1c9d896d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e7606b909d1437fba2b13c1c9d896d2" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 46309 } }
I20260925 22:40:02.140944 24226 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e7606b909d1437fba2b13c1c9d896d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:40:02.141022 24226 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e7606b909d1437fba2b13c1c9d896d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e7606b909d1437fba2b13c1c9d896d2, State: Initialized, Role: FOLLOWER
I20260925 22:40:02.141623 24226 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e7606b909d1437fba2b13c1c9d896d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e7606b909d1437fba2b13c1c9d896d2" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 46309 } }
I20260925 22:40:02.141805 24226 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e7606b909d1437fba2b13c1c9d896d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:40:02.141873 24226 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e7606b909d1437fba2b13c1c9d896d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:40:02.141973 24226 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e7606b909d1437fba2b13c1c9d896d2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:40:02.143185 24226 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e7606b909d1437fba2b13c1c9d896d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e7606b909d1437fba2b13c1c9d896d2" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 46309 } }
I20260925 22:40:02.143520 24226 leader_election.cc:302] T 00000000000000000000000000000000 P 5e7606b909d1437fba2b13c1c9d896d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e7606b909d1437fba2b13c1c9d896d2; no voters:
I20260925 22:40:02.143865 24226 leader_election.cc:288] T 00000000000000000000000000000000 P 5e7606b909d1437fba2b13c1c9d896d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:40:02.144026 24231 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e7606b909d1437fba2b13c1c9d896d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:40:02.144282 24231 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e7606b909d1437fba2b13c1c9d896d2 [term 1 LEADER]: Becoming Leader. State: Replica: 5e7606b909d1437fba2b13c1c9d896d2, State: Running, Role: LEADER
I20260925 22:40:02.144712 24231 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e7606b909d1437fba2b13c1c9d896d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e7606b909d1437fba2b13c1c9d896d2" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 46309 } }
I20260925 22:40:02.145292 24226 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e7606b909d1437fba2b13c1c9d896d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:40:02.146636 24233 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e7606b909d1437fba2b13c1c9d896d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e7606b909d1437fba2b13c1c9d896d2. Latest consensus state: current_term: 1 leader_uuid: "5e7606b909d1437fba2b13c1c9d896d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e7606b909d1437fba2b13c1c9d896d2" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 46309 } } }
I20260925 22:40:02.146785 24233 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e7606b909d1437fba2b13c1c9d896d2 [sys.catalog]: This master's current role is: LEADER
I20260925 22:40:02.146934 24232 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e7606b909d1437fba2b13c1c9d896d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e7606b909d1437fba2b13c1c9d896d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e7606b909d1437fba2b13c1c9d896d2" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 46309 } } }
I20260925 22:40:02.147107 24232 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e7606b909d1437fba2b13c1c9d896d2 [sys.catalog]: This master's current role is: LEADER
I20260925 22:40:02.147421 24240 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:40:02.150636 24240 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:40:02.157124 24240 catalog_manager.cc:1347] Generated new cluster ID: bc91ba09072145429e111ed9a2b122e5
I20260925 22:40:02.157205 24240 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:40:02.185321 24240 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:40:02.186582 24240 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:40:02.197436 24240 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e7606b909d1437fba2b13c1c9d896d2: Generated new TSK 0
I20260925 22:40:02.198356 24240 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.246.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.193.keytab.
Entry for principal kudu/127.6.246.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.193.keytab.
I20260925 22:40:02.236756 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.193:0
--local_ip_for_outbound_sockets=127.6.246.193
--webserver_interface=127.6.246.193
--webserver_port=0
--tserver_master_addrs=127.6.246.254:46309
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.193.keytab
--principal=kudu/127.6.246.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:44821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:40:02.372924 24252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:40:02.373191 24252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:40:02.373235 24252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:40:02.377511 24252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:40:02.377578 24252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:40:02.377671 24252 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.193
I20260925 22:40:02.383117 24252 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:44821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.193.keytab
--principal=kudu/127.6.246.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.246.193
--webserver_port=0
--tserver_master_addrs=127.6.246.254:46309
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24252
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.193
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:40:02.384330 24252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:40:02.385398 24252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:40:02.395133 24257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:02.395133 24258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:02.395224 24260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:40:02.395612 24252 server_base.cc:1034] running on GCE node
I20260925 22:40:02.396029 24252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:40:02.396760 24252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:40:02.397941 24252 hybrid_clock.cc:648] HybridClock initialized: now 1790376002397927 us; error 28 us; skew 500 ppm
Sep 25 22:40:02 dist-test-slave-wjdr krb5kdc[24145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376002, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:40:02.400763 24252 init.cc:377] Logged in from keytab as kudu/127.6.246.193@KRBTEST.COM (short username kudu)
I20260925 22:40:02.401690 24252 webserver.cc:460] Webserver started at http://127.6.246.193:36567/ using document root <none> and password file <none>
I20260925 22:40:02.402290 24252 fs_manager.cc:359] Metadata directory not provided
I20260925 22:40:02.402387 24252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:40:02.402624 24252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:40:02.404757 24252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/minicluster-data/ts-0/data/instance:
uuid: "0320ed92ac80465bbc0134e152994016"
format_stamp: "Formatted at 2026-09-25 22:40:02 on dist-test-slave-wjdr"
server_key: "260e17812400abcb2d0031ae3ff34c96"
server_key_iv: "7019adee4abc99e8569107a52e130533"
server_key_version: "encryptionkey@0"
I20260925 22:40:02.405324 24252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/minicluster-data/ts-0/wal/instance:
uuid: "0320ed92ac80465bbc0134e152994016"
format_stamp: "Formatted at 2026-09-25 22:40:02 on dist-test-slave-wjdr"
server_key: "260e17812400abcb2d0031ae3ff34c96"
server_key_iv: "7019adee4abc99e8569107a52e130533"
server_key_version: "encryptionkey@0"
I20260925 22:40:02.409103 24252 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 22:40:02.411585 24267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:02.412904 24252 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:40:02.413035 24252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/minicluster-data/ts-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/minicluster-data/ts-0/wal
uuid: "0320ed92ac80465bbc0134e152994016"
format_stamp: "Formatted at 2026-09-25 22:40:02 on dist-test-slave-wjdr"
server_key: "260e17812400abcb2d0031ae3ff34c96"
server_key_iv: "7019adee4abc99e8569107a52e130533"
server_key_version: "encryptionkey@0"
I20260925 22:40:02.413144 24252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:40:02.424337 24252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:40:02.427393 24252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:40:02.427609 24252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:40:02.428231 24252 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:40:02.429112 24252 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:40:02.429186 24252 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:02.429270 24252 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:40:02.429322 24252 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:02.438838 24252 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.193:40971
I20260925 22:40:02.438856 24380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.193:40971 every 8 connection(s)
I20260925 22:40:02.439797 24252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
I20260925 22:40:02.445158 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 24252
I20260925 22:40:02.445291 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/minicluster-data/ts-0/wal/instance
I20260925 22:40:02.445626 7131 external_mini_cluster.cc:1428] Setting key 0c243dab0e2a81e1072a1b8415d966bc
Sep 25 22:40:02 dist-test-slave-wjdr krb5kdc[24145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376002, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.193@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
I20260925 22:40:02.455088 24229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:02.441779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.193:37299 (local address 127.6.246.254:46309)
0925 22:40:02.441977 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:02.441981 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:02.442769 (+ 788us) server_negotiation.cc:407] Connection header received
0925 22:40:02.444018 (+ 1249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:02.444021 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:02.444085 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:02.444165 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:02.445500 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:02.446267 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:02.447646 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:02.447877 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:02.447962 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:02.450956 (+ 2994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:02.450963 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:02.450970 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:02.451004 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:02.452894 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:02.453417 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:02.453423 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:02.453427 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:02.453526 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:02.453878 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:02.453883 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:02.453886 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:02.454147 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:02.454251 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:02.454752 (+ 501us) server_negotiation.cc:299] Negotiation successful
0925 22:40:02.454885 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260925 22:40:02.455837 24383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:02.442131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:46309 (local address 127.6.246.193:37299)
0925 22:40:02.442565 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:02.442607 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:02.443806 (+ 1199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:02.444330 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:02.444336 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:02.444744 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:40:02.445328 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:02.445362 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:02.446393 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:02.446397 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:02.447524 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:02.447535 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:02.448037 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:02.448043 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:02.448291 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:02.449007 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:40:02.449034 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:40:02.450779 (+ 1745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:40:02.453015 (+ 2236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:02.453020 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:02.453033 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:02.453307 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:02.453626 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:02.453628 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:02.453630 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:02.453774 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:02.454252 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:40:02.454270 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:40:02.454505 (+ 235us) client_negotiation.cc:769] Sending connection context
0925 22:40:02.454666 (+ 161us) client_negotiation.cc:240] Negotiation successful
0925 22:40:02.454930 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":110,"threads_started":1}
WARNING: no policy specified for kudu/127.6.246.194@KRBTEST.COM; defaulting to no policy
I20260925 22:40:02.457351 24381 heartbeater.cc:347] Connected to a master server at 127.6.246.254:46309
I20260925 22:40:02.457592 24381 heartbeater.cc:464] Registering TS with master...
I20260925 22:40:02.458055 24381 heartbeater.cc:511] Master 127.6.246.254:46309 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.194@KRBTEST.COM" created.
I20260925 22:40:02.459889 24190 ts_manager.cc:194] Registered new tserver with Master: 0320ed92ac80465bbc0134e152994016 (127.6.246.193:40971)
I20260925 22:40:02.461167 24190 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.246.193@KRBTEST.COM'} at 127.6.246.193:37299
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.194.keytab.
Entry for principal kudu/127.6.246.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.194.keytab.
I20260925 22:40:02.476056 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.194:0
--local_ip_for_outbound_sockets=127.6.246.194
--webserver_interface=127.6.246.194
--webserver_port=0
--tserver_master_addrs=127.6.246.254:46309
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.194.keytab
--principal=kudu/127.6.246.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:44821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:40:02.587121 24386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:40:02.587370 24386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:40:02.587414 24386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:40:02.591648 24386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:40:02.591719 24386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:40:02.591781 24386 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.194
I20260925 22:40:02.597122 24386 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:44821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.194.keytab
--principal=kudu/127.6.246.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.246.194
--webserver_port=0
--tserver_master_addrs=127.6.246.254:46309
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24386
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.194
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:40:02.598153 24386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:40:02.599143 24386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:40:02.608527 24391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:02.608522 24392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:02.608579 24394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:40:02.608731 24386 server_base.cc:1034] running on GCE node
I20260925 22:40:02.609480 24386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:40:02.610102 24386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:40:02.611296 24386 hybrid_clock.cc:648] HybridClock initialized: now 1790376002611274 us; error 45 us; skew 500 ppm
Sep 25 22:40:02 dist-test-slave-wjdr krb5kdc[24145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376002, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:40:02.614076 24386 init.cc:377] Logged in from keytab as kudu/127.6.246.194@KRBTEST.COM (short username kudu)
I20260925 22:40:02.614928 24386 webserver.cc:460] Webserver started at http://127.6.246.194:33239/ using document root <none> and password file <none>
I20260925 22:40:02.615507 24386 fs_manager.cc:359] Metadata directory not provided
I20260925 22:40:02.615598 24386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:40:02.615851 24386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:40:02.618028 24386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/minicluster-data/ts-1/data/instance:
uuid: "bf092aa3a22c4119a2da21dc48548058"
format_stamp: "Formatted at 2026-09-25 22:40:02 on dist-test-slave-wjdr"
server_key: "6f6a12db10523cea7b5d6fff305d050f"
server_key_iv: "cfb92b09d8acaeb3a07deda2e2314376"
server_key_version: "encryptionkey@0"
I20260925 22:40:02.618588 24386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/minicluster-data/ts-1/wal/instance:
uuid: "bf092aa3a22c4119a2da21dc48548058"
format_stamp: "Formatted at 2026-09-25 22:40:02 on dist-test-slave-wjdr"
server_key: "6f6a12db10523cea7b5d6fff305d050f"
server_key_iv: "cfb92b09d8acaeb3a07deda2e2314376"
server_key_version: "encryptionkey@0"
I20260925 22:40:02.622555 24386 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:40:02.625155 24401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:02.626400 24386 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:40:02.626557 24386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/minicluster-data/ts-1/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/minicluster-data/ts-1/wal
uuid: "bf092aa3a22c4119a2da21dc48548058"
format_stamp: "Formatted at 2026-09-25 22:40:02 on dist-test-slave-wjdr"
server_key: "6f6a12db10523cea7b5d6fff305d050f"
server_key_iv: "cfb92b09d8acaeb3a07deda2e2314376"
server_key_version: "encryptionkey@0"
I20260925 22:40:02.626685 24386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:40:02.656358 24386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:40:02.659720 24386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:40:02.659967 24386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:40:02.660651 24386 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:40:02.661535 24386 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:40:02.661609 24386 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:02.661681 24386 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:40:02.661734 24386 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:02.671868 24386 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.194:32841
I20260925 22:40:02.671983 24514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.194:32841 every 8 connection(s)
I20260925 22:40:02.672935 24386 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
I20260925 22:40:02.673444 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 24386
I20260925 22:40:02.673563 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/minicluster-data/ts-1/wal/instance
I20260925 22:40:02.673910 7131 external_mini_cluster.cc:1428] Setting key 454038f13a7816c0517745d51a772f25
WARNING: no policy specified for kudu/127.6.246.195@KRBTEST.COM; defaulting to no policy
Sep 25 22:40:02 dist-test-slave-wjdr krb5kdc[24145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376002, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.194@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.195@KRBTEST.COM" created.
I20260925 22:40:02.689152 24229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:02.674988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.194:36191 (local address 127.6.246.254:46309)
0925 22:40:02.675137 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:02.675142 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:02.676149 (+ 1007us) server_negotiation.cc:407] Connection header received
0925 22:40:02.677704 (+ 1555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:02.677707 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:02.677779 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:02.677855 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:02.679187 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:02.679728 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:02.680961 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:02.681202 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:02.681315 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:02.683933 (+ 2618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:02.683936 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:02.683938 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:02.683964 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:02.685424 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:02.685915 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:02.685918 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:02.685920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:02.686045 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:02.687612 (+ 1567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:02.687614 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:02.687616 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:02.687854 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:02.687940 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:02.688694 (+ 754us) server_negotiation.cc:299] Negotiation successful
0925 22:40:02.688887 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 22:40:02.689845 24517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:02.675467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:46309 (local address 127.6.246.194:36191)
0925 22:40:02.675986 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:02.676031 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:02.677495 (+ 1464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:02.678061 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:02.678067 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:02.678499 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:40:02.679045 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:02.679058 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:02.679865 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:02.679868 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:02.680833 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:02.680843 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:02.681355 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:02.681359 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:02.681527 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:02.682180 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:40:02.682197 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:40:02.683764 (+ 1567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:40:02.685523 (+ 1759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:02.685528 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:02.685540 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:02.685792 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:02.686158 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:02.686162 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:02.686165 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:02.686309 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:02.687978 (+ 1669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:40:02.687987 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:40:02.688341 (+ 354us) client_negotiation.cc:769] Sending connection context
0925 22:40:02.688607 (+ 266us) client_negotiation.cc:240] Negotiation successful
0925 22:40:02.688925 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":141,"threads_started":1}
I20260925 22:40:02.691139 24515 heartbeater.cc:347] Connected to a master server at 127.6.246.254:46309
I20260925 22:40:02.691425 24515 heartbeater.cc:464] Registering TS with master...
I20260925 22:40:02.692093 24515 heartbeater.cc:511] Master 127.6.246.254:46309 requested a full tablet report, sending...
I20260925 22:40:02.693277 24190 ts_manager.cc:194] Registered new tserver with Master: bf092aa3a22c4119a2da21dc48548058 (127.6.246.194:32841)
I20260925 22:40:02.693907 24190 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.246.194@KRBTEST.COM'} at 127.6.246.194:36191
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.195.keytab.
Entry for principal kudu/127.6.246.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.195.keytab.
I20260925 22:40:02.703153 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.195:0
--local_ip_for_outbound_sockets=127.6.246.195
--webserver_interface=127.6.246.195
--webserver_port=0
--tserver_master_addrs=127.6.246.254:46309
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.195.keytab
--principal=kudu/127.6.246.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:44821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:40:02.813885 24520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:40:02.814121 24520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:40:02.814163 24520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:40:02.818156 24520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:40:02.818212 24520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:40:02.818271 24520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.195
I20260925 22:40:02.823401 24520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:44821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.195.keytab
--principal=kudu/127.6.246.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.246.195
--webserver_port=0
--tserver_master_addrs=127.6.246.254:46309
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.195
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:40:02.824316 24520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:40:02.825310 24520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:40:02.834614 24525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:02.834654 24526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:02.834631 24528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:40:02.834944 24520 server_base.cc:1034] running on GCE node
I20260925 22:40:02.835314 24520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:40:02.835906 24520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:40:02.837086 24520 hybrid_clock.cc:648] HybridClock initialized: now 1790376002837068 us; error 33 us; skew 500 ppm
Sep 25 22:40:02 dist-test-slave-wjdr krb5kdc[24145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376002, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:40:02.839735 24520 init.cc:377] Logged in from keytab as kudu/127.6.246.195@KRBTEST.COM (short username kudu)
I20260925 22:40:02.840682 24520 webserver.cc:460] Webserver started at http://127.6.246.195:38527/ using document root <none> and password file <none>
I20260925 22:40:02.841291 24520 fs_manager.cc:359] Metadata directory not provided
I20260925 22:40:02.841377 24520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:40:02.841578 24520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:40:02.843537 24520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/minicluster-data/ts-2/data/instance:
uuid: "ab86b6da6f514b01b8119c0de734b967"
format_stamp: "Formatted at 2026-09-25 22:40:02 on dist-test-slave-wjdr"
server_key: "31132c85f53466c6aebd0eaaa6b46f35"
server_key_iv: "0276c177620852631e276c27eb0d42a2"
server_key_version: "encryptionkey@0"
I20260925 22:40:02.844091 24520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/minicluster-data/ts-2/wal/instance:
uuid: "ab86b6da6f514b01b8119c0de734b967"
format_stamp: "Formatted at 2026-09-25 22:40:02 on dist-test-slave-wjdr"
server_key: "31132c85f53466c6aebd0eaaa6b46f35"
server_key_iv: "0276c177620852631e276c27eb0d42a2"
server_key_version: "encryptionkey@0"
I20260925 22:40:02.848007 24520 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260925 22:40:02.850379 24535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:02.851526 24520 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 22:40:02.851656 24520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/minicluster-data/ts-2/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/minicluster-data/ts-2/wal
uuid: "ab86b6da6f514b01b8119c0de734b967"
format_stamp: "Formatted at 2026-09-25 22:40:02 on dist-test-slave-wjdr"
server_key: "31132c85f53466c6aebd0eaaa6b46f35"
server_key_iv: "0276c177620852631e276c27eb0d42a2"
server_key_version: "encryptionkey@0"
I20260925 22:40:02.851801 24520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:40:02.865607 24520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:40:02.868394 24520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:40:02.868618 24520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:40:02.869223 24520 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:40:02.870033 24520 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:40:02.870102 24520 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:02.870172 24520 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:40:02.870214 24520 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:02.879578 24520 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.195:45969
I20260925 22:40:02.879603 24648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.195:45969 every 8 connection(s)
I20260925 22:40:02.880488 24520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
Sep 25 22:40:02 dist-test-slave-wjdr krb5kdc[24145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376002, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.195@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
I20260925 22:40:02.889858 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 24520
I20260925 22:40:02.890012 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/minicluster-data/ts-2/wal/instance
I20260925 22:40:02.890398 7131 external_mini_cluster.cc:1428] Setting key 1b3906afdf1e4cec849724808c9e451f
I20260925 22:40:02.893875 24229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:02.882319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.195:52017 (local address 127.6.246.254:46309)
0925 22:40:02.882474 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:02.882479 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:02.883321 (+ 842us) server_negotiation.cc:407] Connection header received
0925 22:40:02.884573 (+ 1252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:02.884576 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:02.884622 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:02.884696 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:02.885822 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:02.886358 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:02.887452 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:02.887652 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:02.887726 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:02.890192 (+ 2466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:02.890197 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:02.890200 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:02.890230 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:02.891858 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:02.892432 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:02.892439 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:02.892443 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:02.892565 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:02.892912 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:02.892915 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:02.892916 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:02.893086 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:02.893166 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:02.893627 (+ 461us) server_negotiation.cc:299] Negotiation successful
0925 22:40:02.893758 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 22:40:02.894701 24651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:02.882715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:46309 (local address 127.6.246.195:52017)
0925 22:40:02.883157 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:02.883197 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:02.884324 (+ 1127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:02.884856 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:02.884862 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:02.885205 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:40:02.885676 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:02.885689 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:02.886476 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:02.886479 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:02.887348 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:02.887357 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:02.887781 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:02.887784 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:02.887919 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:02.888483 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:40:02.888499 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:40:02.889998 (+ 1499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:40:02.891962 (+ 1964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:02.891968 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:02.891981 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:02.892304 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:02.892669 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:02.892672 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:02.892674 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:02.892824 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:02.893203 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:40:02.893208 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:40:02.893424 (+ 216us) client_negotiation.cc:769] Sending connection context
0925 22:40:02.893578 (+ 154us) client_negotiation.cc:240] Negotiation successful
0925 22:40:02.893861 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":103,"threads_started":1}
I20260925 22:40:02.895766 24649 heartbeater.cc:347] Connected to a master server at 127.6.246.254:46309
I20260925 22:40:02.896003 24649 heartbeater.cc:464] Registering TS with master...
I20260925 22:40:02.896540 24649 heartbeater.cc:511] Master 127.6.246.254:46309 requested a full tablet report, sending...
I20260925 22:40:02.897522 24190 ts_manager.cc:194] Registered new tserver with Master: ab86b6da6f514b01b8119c0de734b967 (127.6.246.195:45969)
I20260925 22:40:02.898108 24190 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.246.195@KRBTEST.COM'} at 127.6.246.195:52017
I20260925 22:40:02.904285 7131 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:40:02.914470 24229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:02.906375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38124 (local address 127.6.246.254:46309)
0925 22:40:02.906565 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:02.906570 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:02.906729 (+ 159us) server_negotiation.cc:407] Connection header received
0925 22:40:02.906899 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:02.906902 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:02.906954 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:02.907039 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:02.907800 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:02.908329 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:02.909331 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:02.909557 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:02.909662 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:02.911007 (+ 1345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:02.911011 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:02.911013 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:02.911039 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:02.912312 (+ 1273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:02.912795 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:02.912798 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:02.912800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:02.912893 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:02.913179 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:02.913182 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:02.913184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:02.913405 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:02.913549 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:02.914259 (+ 710us) server_negotiation.cc:299] Negotiation successful
0925 22:40:02.914355 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 22:40:02.917474 24190 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38124:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:40:02.920583 24190 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:40:02.934832 24662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:02.928694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.193:40971 (local address 127.0.0.1:43080)
0925 22:40:02.930147 (+ 1453us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:02.930195 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:02.930345 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:02.931089 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:02.931094 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:02.931122 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:40:02.931297 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:02.931304 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:02.932420 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:02.932423 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:02.933465 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:02.933474 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:02.934300 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:02.934303 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:02.934416 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:02.934452 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 22:40:02.934565 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 22:40:02.934683 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1282,"mutex_wait_us":47,"spinlock_wait_cycles":2816,"thread_start_us":1194,"threads_started":1}
I20260925 22:40:02.934844 24661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:02.928819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43080 (local address 127.6.246.193:40971)
0925 22:40:02.930108 (+ 1289us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:02.930123 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:02.930295 (+ 172us) server_negotiation.cc:407] Connection header received
0925 22:40:02.930516 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:02.930522 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:02.930723 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:02.930909 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:40:02.931412 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:02.932264 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:02.933594 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:02.934172 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:02.934294 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:02.934445 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:02.934593 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 22:40:02.934684 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1167,"thread_start_us":80,"threads_started":1}
I20260925 22:40:02.935921 24660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:02.929071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.195:45969 (local address 127.0.0.1:52642)
0925 22:40:02.930173 (+ 1102us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:02.930207 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:02.930345 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:02.931091 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:02.931096 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:02.931130 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:40:02.931317 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:02.931324 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:02.932816 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:02.932819 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:02.934166 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:02.934177 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:02.935481 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:02.935485 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:02.935578 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:02.935599 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 22:40:02.935675 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 22:40:02.935749 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1050,"mutex_wait_us":85,"spinlock_wait_cycles":3456,"thread_start_us":97,"threads_started":1}
I20260925 22:40:02.936055 24664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:02.929578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52642 (local address 127.6.246.195:45969)
0925 22:40:02.930212 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:02.930227 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:02.930317 (+ 90us) server_negotiation.cc:407] Connection header received
0925 22:40:02.930520 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:02.930525 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:02.930729 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:02.930875 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:40:02.931466 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:02.932660 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:02.934312 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:02.935113 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:02.935224 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:02.935343 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:02.935721 (+ 378us) server_negotiation.cc:299] Negotiation successful
0925 22:40:02.935916 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":162,"threads_started":1}
I20260925 22:40:02.936435 24665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:02.928796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.194:32841 (local address 127.0.0.1:49502)
0925 22:40:02.930524 (+ 1728us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:02.930536 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:02.930614 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:02.931180 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:02.931183 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:02.931204 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:40:02.931324 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:02.931330 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:02.932816 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:02.932819 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:02.934166 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:02.934177 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:02.936051 (+ 1874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:02.936056 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:02.936140 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:02.936163 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 22:40:02.936235 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 22:40:02.936306 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1232,"mutex_wait_us":136,"spinlock_wait_cycles":7424,"thread_start_us":96,"threads_started":1}
I20260925 22:40:02.936861 24663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:02.929578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49502 (local address 127.6.246.194:32841)
0925 22:40:02.930016 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:02.930028 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:02.930631 (+ 603us) server_negotiation.cc:407] Connection header received
0925 22:40:02.930721 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:02.930725 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:02.930892 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:02.931023 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:40:02.931458 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:02.932660 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:02.934312 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:02.935113 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:02.935224 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:02.935343 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:02.936617 (+ 1274us) server_negotiation.cc:299] Negotiation successful
0925 22:40:02.936708 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":128,"threads_started":1}
I20260925 22:40:02.938678 24583 tablet_service.cc:1467] Processing CreateTablet for tablet dc089db06e6e48999cb2346a2fc2edcb (DEFAULT_TABLE table=test-table [id=c555eb2cf07c4dd38fb373a76ae8db3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:40:02.938678 24315 tablet_service.cc:1467] Processing CreateTablet for tablet dc089db06e6e48999cb2346a2fc2edcb (DEFAULT_TABLE table=test-table [id=c555eb2cf07c4dd38fb373a76ae8db3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:40:02.938839 24449 tablet_service.cc:1467] Processing CreateTablet for tablet dc089db06e6e48999cb2346a2fc2edcb (DEFAULT_TABLE table=test-table [id=c555eb2cf07c4dd38fb373a76ae8db3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:40:02.939823 24583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc089db06e6e48999cb2346a2fc2edcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:40:02.939817 24315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc089db06e6e48999cb2346a2fc2edcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:40:02.939822 24449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc089db06e6e48999cb2346a2fc2edcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:40:02.946063 24666 tablet_bootstrap.cc:492] T dc089db06e6e48999cb2346a2fc2edcb P bf092aa3a22c4119a2da21dc48548058: Bootstrap starting.
I20260925 22:40:02.947417 24668 tablet_bootstrap.cc:492] T dc089db06e6e48999cb2346a2fc2edcb P 0320ed92ac80465bbc0134e152994016: Bootstrap starting.
I20260925 22:40:02.948523 24666 tablet_bootstrap.cc:654] T dc089db06e6e48999cb2346a2fc2edcb P bf092aa3a22c4119a2da21dc48548058: Neither blocks nor log segments found. Creating new log.
I20260925 22:40:02.948787 24667 tablet_bootstrap.cc:492] T dc089db06e6e48999cb2346a2fc2edcb P ab86b6da6f514b01b8119c0de734b967: Bootstrap starting.
I20260925 22:40:02.949150 24668 tablet_bootstrap.cc:654] T dc089db06e6e48999cb2346a2fc2edcb P 0320ed92ac80465bbc0134e152994016: Neither blocks nor log segments found. Creating new log.
I20260925 22:40:02.949975 24666 log.cc:824] T dc089db06e6e48999cb2346a2fc2edcb P bf092aa3a22c4119a2da21dc48548058: Log is configured to *not* fsync() on all Append() calls
I20260925 22:40:02.950276 24668 log.cc:824] T dc089db06e6e48999cb2346a2fc2edcb P 0320ed92ac80465bbc0134e152994016: Log is configured to *not* fsync() on all Append() calls
I20260925 22:40:02.950702 24667 tablet_bootstrap.cc:654] T dc089db06e6e48999cb2346a2fc2edcb P ab86b6da6f514b01b8119c0de734b967: Neither blocks nor log segments found. Creating new log.
I20260925 22:40:02.951668 24667 log.cc:824] T dc089db06e6e48999cb2346a2fc2edcb P ab86b6da6f514b01b8119c0de734b967: Log is configured to *not* fsync() on all Append() calls
I20260925 22:40:02.952056 24668 tablet_bootstrap.cc:492] T dc089db06e6e48999cb2346a2fc2edcb P 0320ed92ac80465bbc0134e152994016: No bootstrap required, opened a new log
I20260925 22:40:02.952260 24668 ts_tablet_manager.cc:1397] T dc089db06e6e48999cb2346a2fc2edcb P 0320ed92ac80465bbc0134e152994016: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 22:40:02.953022 24666 tablet_bootstrap.cc:492] T dc089db06e6e48999cb2346a2fc2edcb P bf092aa3a22c4119a2da21dc48548058: No bootstrap required, opened a new log
I20260925 22:40:02.953198 24666 ts_tablet_manager.cc:1397] T dc089db06e6e48999cb2346a2fc2edcb P bf092aa3a22c4119a2da21dc48548058: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20260925 22:40:02.953533 24667 tablet_bootstrap.cc:492] T dc089db06e6e48999cb2346a2fc2edcb P ab86b6da6f514b01b8119c0de734b967: No bootstrap required, opened a new log
I20260925 22:40:02.953702 24667 ts_tablet_manager.cc:1397] T dc089db06e6e48999cb2346a2fc2edcb P ab86b6da6f514b01b8119c0de734b967: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 22:40:02.955536 24668 raft_consensus.cc:348] T dc089db06e6e48999cb2346a2fc2edcb P 0320ed92ac80465bbc0134e152994016 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0320ed92ac80465bbc0134e152994016" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 40971 } } peers { permanent_uuid: "bf092aa3a22c4119a2da21dc48548058" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 32841 } } peers { permanent_uuid: "ab86b6da6f514b01b8119c0de734b967" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 45969 } }
I20260925 22:40:02.955766 24668 raft_consensus.cc:374] T dc089db06e6e48999cb2346a2fc2edcb P 0320ed92ac80465bbc0134e152994016 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:40:02.955847 24668 raft_consensus.cc:729] T dc089db06e6e48999cb2346a2fc2edcb P 0320ed92ac80465bbc0134e152994016 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0320ed92ac80465bbc0134e152994016, State: Initialized, Role: FOLLOWER
I20260925 22:40:02.956364 24668 consensus_queue.cc:260] T dc089db06e6e48999cb2346a2fc2edcb P 0320ed92ac80465bbc0134e152994016 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0320ed92ac80465bbc0134e152994016" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 40971 } } peers { permanent_uuid: "bf092aa3a22c4119a2da21dc48548058" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 32841 } } peers { permanent_uuid: "ab86b6da6f514b01b8119c0de734b967" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 45969 } }
I20260925 22:40:02.957047 24666 raft_consensus.cc:348] T dc089db06e6e48999cb2346a2fc2edcb P bf092aa3a22c4119a2da21dc48548058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0320ed92ac80465bbc0134e152994016" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 40971 } } peers { permanent_uuid: "bf092aa3a22c4119a2da21dc48548058" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 32841 } } peers { permanent_uuid: "ab86b6da6f514b01b8119c0de734b967" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 45969 } }
I20260925 22:40:02.957371 24666 raft_consensus.cc:374] T dc089db06e6e48999cb2346a2fc2edcb P bf092aa3a22c4119a2da21dc48548058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:40:02.957439 24666 raft_consensus.cc:729] T dc089db06e6e48999cb2346a2fc2edcb P bf092aa3a22c4119a2da21dc48548058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf092aa3a22c4119a2da21dc48548058, State: Initialized, Role: FOLLOWER
I20260925 22:40:02.957309 24667 raft_consensus.cc:348] T dc089db06e6e48999cb2346a2fc2edcb P ab86b6da6f514b01b8119c0de734b967 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0320ed92ac80465bbc0134e152994016" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 40971 } } peers { permanent_uuid: "bf092aa3a22c4119a2da21dc48548058" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 32841 } } peers { permanent_uuid: "ab86b6da6f514b01b8119c0de734b967" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 45969 } }
I20260925 22:40:02.957540 24668 ts_tablet_manager.cc:1428] T dc089db06e6e48999cb2346a2fc2edcb P 0320ed92ac80465bbc0134e152994016: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260925 22:40:02.957561 24667 raft_consensus.cc:374] T dc089db06e6e48999cb2346a2fc2edcb P ab86b6da6f514b01b8119c0de734b967 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:40:02.957605 24667 raft_consensus.cc:729] T dc089db06e6e48999cb2346a2fc2edcb P ab86b6da6f514b01b8119c0de734b967 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab86b6da6f514b01b8119c0de734b967, State: Initialized, Role: FOLLOWER
I20260925 22:40:02.957913 24381 heartbeater.cc:503] Master 127.6.246.254:46309 was elected leader, sending a full tablet report...
I20260925 22:40:02.957916 24666 consensus_queue.cc:260] T dc089db06e6e48999cb2346a2fc2edcb P bf092aa3a22c4119a2da21dc48548058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0320ed92ac80465bbc0134e152994016" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 40971 } } peers { permanent_uuid: "bf092aa3a22c4119a2da21dc48548058" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 32841 } } peers { permanent_uuid: "ab86b6da6f514b01b8119c0de734b967" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 45969 } }
I20260925 22:40:02.957989 24667 consensus_queue.cc:260] T dc089db06e6e48999cb2346a2fc2edcb P ab86b6da6f514b01b8119c0de734b967 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0320ed92ac80465bbc0134e152994016" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 40971 } } peers { permanent_uuid: "bf092aa3a22c4119a2da21dc48548058" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 32841 } } peers { permanent_uuid: "ab86b6da6f514b01b8119c0de734b967" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 45969 } }
I20260925 22:40:02.958554 24515 heartbeater.cc:503] Master 127.6.246.254:46309 was elected leader, sending a full tablet report...
I20260925 22:40:02.958839 24667 ts_tablet_manager.cc:1428] T dc089db06e6e48999cb2346a2fc2edcb P ab86b6da6f514b01b8119c0de734b967: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260925 22:40:02.958832 24666 ts_tablet_manager.cc:1428] T dc089db06e6e48999cb2346a2fc2edcb P bf092aa3a22c4119a2da21dc48548058: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 22:40:02.959125 24649 heartbeater.cc:503] Master 127.6.246.254:46309 was elected leader, sending a full tablet report...
I20260925 22:40:03.121906 24674 raft_consensus.cc:482] T dc089db06e6e48999cb2346a2fc2edcb P ab86b6da6f514b01b8119c0de734b967 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:40:03.122124 24674 raft_consensus.cc:504] T dc089db06e6e48999cb2346a2fc2edcb P ab86b6da6f514b01b8119c0de734b967 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0320ed92ac80465bbc0134e152994016" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 40971 } } peers { permanent_uuid: "bf092aa3a22c4119a2da21dc48548058" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 32841 } } peers { permanent_uuid: "ab86b6da6f514b01b8119c0de734b967" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 45969 } }
I20260925 22:40:03.123383 24674 leader_election.cc:288] T dc089db06e6e48999cb2346a2fc2edcb P ab86b6da6f514b01b8119c0de734b967 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0320ed92ac80465bbc0134e152994016 (127.6.246.193:40971), bf092aa3a22c4119a2da21dc48548058 (127.6.246.194:32841)
I20260925 22:40:03.127422 24672 raft_consensus.cc:482] T dc089db06e6e48999cb2346a2fc2edcb P 0320ed92ac80465bbc0134e152994016 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:40:03.127606 24672 raft_consensus.cc:504] T dc089db06e6e48999cb2346a2fc2edcb P 0320ed92ac80465bbc0134e152994016 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0320ed92ac80465bbc0134e152994016" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 40971 } } peers { permanent_uuid: "bf092aa3a22c4119a2da21dc48548058" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 32841 } } peers { permanent_uuid: "ab86b6da6f514b01b8119c0de734b967" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 45969 } }
I20260925 22:40:03.127876 24651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:03.123610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.193:40971 (local address 127.6.246.195:42031)
0925 22:40:03.123826 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:03.123843 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:03.123937 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:03.124353 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:03.124358 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:03.124377 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:40:03.124547 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:03.124554 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:03.125388 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:03.125395 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:03.126446 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:03.126456 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:03.127458 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:03.127461 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:03.127577 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:03.127599 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 22:40:03.127670 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 22:40:03.127742 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107}
I20260925 22:40:03.128260 24663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:03.123905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.195:42719 (local address 127.6.246.194:32841)
0925 22:40:03.124021 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:03.124026 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:03.124271 (+ 245us) server_negotiation.cc:407] Connection header received
0925 22:40:03.124418 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:03.124421 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:03.124489 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:03.124545 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:40:03.124946 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:03.125709 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:03.126972 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:03.127588 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:03.127703 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:03.127751 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:03.128074 (+ 323us) server_negotiation.cc:299] Negotiation successful
0925 22:40:03.128148 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 22:40:03.128311 24661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:03.123996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.195:42031 (local address 127.6.246.193:40971)
0925 22:40:03.124123 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:03.124127 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:03.124141 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:40:03.124194 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:03.124197 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:03.124254 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:03.124332 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:40:03.124679 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:03.125267 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:03.126582 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:03.127188 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:03.127320 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:03.127373 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:03.128118 (+ 745us) server_negotiation.cc:299] Negotiation successful
0925 22:40:03.128193 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 22:40:03.128770 24675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:03.123649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.194:32841 (local address 127.6.246.195:42719)
0925 22:40:03.124173 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:03.124183 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:03.124294 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:03.124683 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:03.124686 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:03.124701 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:40:03.124828 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:03.124834 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:03.125878 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:03.125882 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:03.126856 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:03.126863 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:03.127824 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:03.127828 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:03.127927 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:03.127950 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 22:40:03.128387 (+ 437us) client_negotiation.cc:240] Negotiation successful
0925 22:40:03.128633 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":130,"threads_started":1}
I20260925 22:40:03.128899 24335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc089db06e6e48999cb2346a2fc2edcb" candidate_uuid: "ab86b6da6f514b01b8119c0de734b967" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0320ed92ac80465bbc0134e152994016" is_pre_election: true
I20260925 22:40:03.129105 24335 raft_consensus.cc:2454] T dc089db06e6e48999cb2346a2fc2edcb P 0320ed92ac80465bbc0134e152994016 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab86b6da6f514b01b8119c0de734b967 in term 0.
I20260925 22:40:03.129410 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc089db06e6e48999cb2346a2fc2edcb" candidate_uuid: "ab86b6da6f514b01b8119c0de734b967" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf092aa3a22c4119a2da21dc48548058" is_pre_election: true
I20260925 22:40:03.129506 24672 leader_election.cc:288] T dc089db06e6e48999cb2346a2fc2edcb P 0320ed92ac80465bbc0134e152994016 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf092aa3a22c4119a2da21dc48548058 (127.6.246.194:32841), ab86b6da6f514b01b8119c0de734b967 (127.6.246.195:45969)
I20260925 22:40:03.129621 24537 leader_election.cc:302] T dc089db06e6e48999cb2346a2fc2edcb P ab86b6da6f514b01b8119c0de734b967 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0320ed92ac80465bbc0134e152994016, ab86b6da6f514b01b8119c0de734b967; no voters:
I20260925 22:40:03.129683 24469 raft_consensus.cc:2454] T dc089db06e6e48999cb2346a2fc2edcb P bf092aa3a22c4119a2da21dc48548058 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab86b6da6f514b01b8119c0de734b967 in term 0.
I20260925 22:40:03.129910 24674 raft_consensus.cc:2781] T dc089db06e6e48999cb2346a2fc2edcb P ab86b6da6f514b01b8119c0de734b967 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:40:03.129980 24674 raft_consensus.cc:482] T dc089db06e6e48999cb2346a2fc2edcb P ab86b6da6f514b01b8119c0de734b967 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:40:03.130017 24674 raft_consensus.cc:3037] T dc089db06e6e48999cb2346a2fc2edcb P ab86b6da6f514b01b8119c0de734b967 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:40:03.131371 24674 raft_consensus.cc:504] T dc089db06e6e48999cb2346a2fc2edcb P ab86b6da6f514b01b8119c0de734b967 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0320ed92ac80465bbc0134e152994016" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 40971 } } peers { permanent_uuid: "bf092aa3a22c4119a2da21dc48548058" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 32841 } } peers { permanent_uuid: "ab86b6da6f514b01b8119c0de734b967" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 45969 } }
W20260925 22:40:03.131513 24650 tablet.cc:2369] T dc089db06e6e48999cb2346a2fc2edcb P ab86b6da6f514b01b8119c0de734b967: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:40:03.131711 24674 leader_election.cc:288] T dc089db06e6e48999cb2346a2fc2edcb P ab86b6da6f514b01b8119c0de734b967 [CANDIDATE]: Term 1 election: Requested vote from peers 0320ed92ac80465bbc0134e152994016 (127.6.246.193:40971), bf092aa3a22c4119a2da21dc48548058 (127.6.246.194:32841)
I20260925 22:40:03.132318 24335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc089db06e6e48999cb2346a2fc2edcb" candidate_uuid: "ab86b6da6f514b01b8119c0de734b967" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0320ed92ac80465bbc0134e152994016"
I20260925 22:40:03.132318 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc089db06e6e48999cb2346a2fc2edcb" candidate_uuid: "ab86b6da6f514b01b8119c0de734b967" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf092aa3a22c4119a2da21dc48548058"
I20260925 22:40:03.132432 24335 raft_consensus.cc:3037] T dc089db06e6e48999cb2346a2fc2edcb P 0320ed92ac80465bbc0134e152994016 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:40:03.132432 24469 raft_consensus.cc:3037] T dc089db06e6e48999cb2346a2fc2edcb P bf092aa3a22c4119a2da21dc48548058 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:40:03.133739 24469 raft_consensus.cc:2454] T dc089db06e6e48999cb2346a2fc2edcb P bf092aa3a22c4119a2da21dc48548058 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab86b6da6f514b01b8119c0de734b967 in term 1.
I20260925 22:40:03.133733 24335 raft_consensus.cc:2454] T dc089db06e6e48999cb2346a2fc2edcb P 0320ed92ac80465bbc0134e152994016 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab86b6da6f514b01b8119c0de734b967 in term 1.
I20260925 22:40:03.134085 24538 leader_election.cc:302] T dc089db06e6e48999cb2346a2fc2edcb P ab86b6da6f514b01b8119c0de734b967 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab86b6da6f514b01b8119c0de734b967, bf092aa3a22c4119a2da21dc48548058; no voters:
I20260925 22:40:03.134322 24674 raft_consensus.cc:2781] T dc089db06e6e48999cb2346a2fc2edcb P ab86b6da6f514b01b8119c0de734b967 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:40:03.134575 24674 raft_consensus.cc:686] T dc089db06e6e48999cb2346a2fc2edcb P ab86b6da6f514b01b8119c0de734b967 [term 1 LEADER]: Becoming Leader. State: Replica: ab86b6da6f514b01b8119c0de734b967, State: Running, Role: LEADER
I20260925 22:40:03.134737 24676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:03.129791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.194:32841 (local address 127.6.246.193:46139)
0925 22:40:03.130272 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:03.130287 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:03.130383 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:03.131009 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:03.131012 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:03.131032 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:40:03.131143 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:03.131151 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:03.132135 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:03.132138 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:03.133379 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:03.133388 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:03.134338 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:03.134342 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:03.134427 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:03.134447 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 22:40:03.134517 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 22:40:03.134579 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"mutex_wait_us":11,"thread_start_us":119,"threads_started":1}
I20260925 22:40:03.134742 24663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:03.130554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.193:46139 (local address 127.6.246.194:32841)
0925 22:40:03.130678 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:03.130682 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:03.130696 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:40:03.130741 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:03.130747 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:03.130791 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:03.130896 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:40:03.131317 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:03.132012 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:03.133514 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:03.134122 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:03.134202 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:03.134247 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:03.134556 (+ 309us) server_negotiation.cc:299] Negotiation successful
0925 22:40:03.134640 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 22:40:03.134953 24674 consensus_queue.cc:237] T dc089db06e6e48999cb2346a2fc2edcb P ab86b6da6f514b01b8119c0de734b967 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0320ed92ac80465bbc0134e152994016" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 40971 } } peers { permanent_uuid: "bf092aa3a22c4119a2da21dc48548058" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 32841 } } peers { permanent_uuid: "ab86b6da6f514b01b8119c0de734b967" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 45969 } }
I20260925 22:40:03.135128 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc089db06e6e48999cb2346a2fc2edcb" candidate_uuid: "0320ed92ac80465bbc0134e152994016" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf092aa3a22c4119a2da21dc48548058" is_pre_election: true
I20260925 22:40:03.135298 24469 raft_consensus.cc:2379] T dc089db06e6e48999cb2346a2fc2edcb P bf092aa3a22c4119a2da21dc48548058 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0320ed92ac80465bbc0134e152994016 in current term 1: Already voted for candidate ab86b6da6f514b01b8119c0de734b967 in this term.
I20260925 22:40:03.135556 24664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:03.131211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.193:33439 (local address 127.6.246.195:45969)
0925 22:40:03.131334 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:03.131340 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:03.131356 (+ 16us) server_negotiation.cc:407] Connection header received
0925 22:40:03.131466 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:03.131469 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:03.131510 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:03.131553 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:40:03.131889 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:03.132637 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:03.134066 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:03.134669 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:03.134755 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:03.134809 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:03.135393 (+ 584us) server_negotiation.cc:299] Negotiation successful
0925 22:40:03.135461 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 22:40:03.135905 24677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:03.130810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.195:45969 (local address 127.6.246.193:33439)
0925 22:40:03.131253 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:03.131265 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:03.131352 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:03.131665 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:03.131667 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:03.131682 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:40:03.131781 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:03.131785 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:03.132766 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:03.132769 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:03.133936 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:03.133945 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:03.135128 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:03.135131 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:03.135232 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:03.135256 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 22:40:03.135724 (+ 468us) client_negotiation.cc:240] Negotiation successful
0925 22:40:03.135805 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":99,"threads_started":1}
I20260925 22:40:03.136368 24603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc089db06e6e48999cb2346a2fc2edcb" candidate_uuid: "0320ed92ac80465bbc0134e152994016" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab86b6da6f514b01b8119c0de734b967" is_pre_election: true
I20260925 22:40:03.136966 24271 leader_election.cc:302] T dc089db06e6e48999cb2346a2fc2edcb P 0320ed92ac80465bbc0134e152994016 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0320ed92ac80465bbc0134e152994016; no voters: ab86b6da6f514b01b8119c0de734b967, bf092aa3a22c4119a2da21dc48548058
I20260925 22:40:03.137238 24672 raft_consensus.cc:2726] T dc089db06e6e48999cb2346a2fc2edcb P 0320ed92ac80465bbc0134e152994016 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 22:40:03.138429 24189 catalog_manager.cc:5495] T dc089db06e6e48999cb2346a2fc2edcb P ab86b6da6f514b01b8119c0de734b967 reported cstate change: term changed from 0 to 1, leader changed from <none> to ab86b6da6f514b01b8119c0de734b967 (127.6.246.195). New cstate: current_term: 1 leader_uuid: "ab86b6da6f514b01b8119c0de734b967" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0320ed92ac80465bbc0134e152994016" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 40971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf092aa3a22c4119a2da21dc48548058" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 32841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab86b6da6f514b01b8119c0de734b967" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 45969 } health_report { overall_health: HEALTHY } } }
I20260925 22:40:03.169905 24664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:03.165827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52652 (local address 127.6.246.195:45969)
0925 22:40:03.165994 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:03.165998 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:03.166009 (+ 11us) server_negotiation.cc:407] Connection header received
0925 22:40:03.166080 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:03.166082 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:03.166132 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:03.166268 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:40:03.166531 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:03.167268 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:03.168299 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:03.168584 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:03.168694 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:03.169060 (+ 366us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:40:03.169490 (+ 430us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:40:03.169620 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:03.169687 (+ 67us) server_negotiation.cc:299] Negotiation successful
0925 22:40:03.169736 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260925 22:40:03.175076 24516 tablet.cc:2369] T dc089db06e6e48999cb2346a2fc2edcb P bf092aa3a22c4119a2da21dc48548058: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:40:03.175607 24469 raft_consensus.cc:1261] T dc089db06e6e48999cb2346a2fc2edcb P bf092aa3a22c4119a2da21dc48548058 [term 1 FOLLOWER]: Refusing update from remote peer ab86b6da6f514b01b8119c0de734b967: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:40:03.175644 24335 raft_consensus.cc:1261] T dc089db06e6e48999cb2346a2fc2edcb P 0320ed92ac80465bbc0134e152994016 [term 1 FOLLOWER]: Refusing update from remote peer ab86b6da6f514b01b8119c0de734b967: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:40:03.176281 24678 consensus_queue.cc:1035] T dc089db06e6e48999cb2346a2fc2edcb P ab86b6da6f514b01b8119c0de734b967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf092aa3a22c4119a2da21dc48548058" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 32841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:40:03.176476 24674 consensus_queue.cc:1035] T dc089db06e6e48999cb2346a2fc2edcb P ab86b6da6f514b01b8119c0de734b967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0320ed92ac80465bbc0134e152994016" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 40971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:40:03.183853 24683 mvcc.cc:205] Tried to move back new op lower bound from 7333380109000400896 to 7333380108843655168. Current Snapshot: MvccSnapshot[applied={T|T < 7333380109000400896}]
I20260925 22:40:03.185791 24682 mvcc.cc:205] Tried to move back new op lower bound from 7333380109000400896 to 7333380108843655168. Current Snapshot: MvccSnapshot[applied={T|T < 7333380109000400896}]
I20260925 22:40:03.191133 24189 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38124:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:40:03.191335 24189 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38124:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:40:03.193856 24189 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5e7606b909d1437fba2b13c1c9d896d2: Sending DeleteTablet for 3 replicas of tablet dc089db06e6e48999cb2346a2fc2edcb
I20260925 22:40:03.194646 24315 tablet_service.cc:1514] Processing DeleteTablet for tablet dc089db06e6e48999cb2346a2fc2edcb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:40:03 UTC) from {username='kudu', principal='kudu/127.6.246.254@KRBTEST.COM'} at 127.0.0.1:43080
I20260925 22:40:03.194684 24583 tablet_service.cc:1514] Processing DeleteTablet for tablet dc089db06e6e48999cb2346a2fc2edcb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:40:03 UTC) from {username='kudu', principal='kudu/127.6.246.254@KRBTEST.COM'} at 127.0.0.1:52642
I20260925 22:40:03.194865 24449 tablet_service.cc:1514] Processing DeleteTablet for tablet dc089db06e6e48999cb2346a2fc2edcb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:40:03 UTC) from {username='kudu', principal='kudu/127.6.246.254@KRBTEST.COM'} at 127.0.0.1:49502
I20260925 22:40:03.195147 24691 tablet_replica.cc:330] T dc089db06e6e48999cb2346a2fc2edcb P 0320ed92ac80465bbc0134e152994016: stopping tablet replica
I20260925 22:40:03.195324 24693 tablet_replica.cc:330] T dc089db06e6e48999cb2346a2fc2edcb P bf092aa3a22c4119a2da21dc48548058: stopping tablet replica
I20260925 22:40:03.195497 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 24252
I20260925 22:40:03.196661 24692 tablet_replica.cc:330] T dc089db06e6e48999cb2346a2fc2edcb P ab86b6da6f514b01b8119c0de734b967: stopping tablet replica
I20260925 22:40:03.196842 24693 raft_consensus.cc:2229] T dc089db06e6e48999cb2346a2fc2edcb P bf092aa3a22c4119a2da21dc48548058 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:40:03.196964 24692 raft_consensus.cc:2229] T dc089db06e6e48999cb2346a2fc2edcb P ab86b6da6f514b01b8119c0de734b967 [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:40:03.197261 24693 raft_consensus.cc:2258] T dc089db06e6e48999cb2346a2fc2edcb P bf092aa3a22c4119a2da21dc48548058 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:40:03.197335 24692 raft_consensus.cc:2258] T dc089db06e6e48999cb2346a2fc2edcb P ab86b6da6f514b01b8119c0de734b967 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:40:03.198750 24692 ts_tablet_manager.cc:1905] T dc089db06e6e48999cb2346a2fc2edcb P ab86b6da6f514b01b8119c0de734b967: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:40:03.199116 24693 ts_tablet_manager.cc:1905] T dc089db06e6e48999cb2346a2fc2edcb P bf092aa3a22c4119a2da21dc48548058: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:40:03.201661 24692 ts_tablet_manager.cc:1918] T dc089db06e6e48999cb2346a2fc2edcb P ab86b6da6f514b01b8119c0de734b967: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:40:03.201808 24693 ts_tablet_manager.cc:1918] T dc089db06e6e48999cb2346a2fc2edcb P bf092aa3a22c4119a2da21dc48548058: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:40:03.201849 24692 log.cc:1196] T dc089db06e6e48999cb2346a2fc2edcb P ab86b6da6f514b01b8119c0de734b967: Deleting WAL directory at /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/minicluster-data/ts-2/wal/wals/dc089db06e6e48999cb2346a2fc2edcb
I20260925 22:40:03.201983 24693 log.cc:1196] T dc089db06e6e48999cb2346a2fc2edcb P bf092aa3a22c4119a2da21dc48548058: Deleting WAL directory at /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790375752797504-7131-0/minicluster-data/ts-1/wal/wals/dc089db06e6e48999cb2346a2fc2edcb
I20260925 22:40:03.202195 24692 ts_tablet_manager.cc:1939] T dc089db06e6e48999cb2346a2fc2edcb P ab86b6da6f514b01b8119c0de734b967: Deleting consensus metadata
I20260925 22:40:03.202370 24693 ts_tablet_manager.cc:1939] T dc089db06e6e48999cb2346a2fc2edcb P bf092aa3a22c4119a2da21dc48548058: Deleting consensus metadata
I20260925 22:40:03.203460 24177 catalog_manager.cc:4841] TS ab86b6da6f514b01b8119c0de734b967 (127.6.246.195:45969): tablet dc089db06e6e48999cb2346a2fc2edcb (table test-table [id=c555eb2cf07c4dd38fb373a76ae8db3e]) successfully deleted
I20260925 22:40:03.203745 24176 catalog_manager.cc:4841] TS bf092aa3a22c4119a2da21dc48548058 (127.6.246.194:32841): tablet dc089db06e6e48999cb2346a2fc2edcb (table test-table [id=c555eb2cf07c4dd38fb373a76ae8db3e]) successfully deleted
W20260925 22:40:03.204200 24175 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.246.193:40971 (error 108)
I20260925 22:40:03.204414 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 24386
I20260925 22:40:03.205854 24665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:03.205586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.193:40971 (local address 127.0.0.1:43082)
0925 22:40:03.205709 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:03.205785 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.193:40971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 22:40:03.206036 24175 catalog_manager.cc:4584] TS 0320ed92ac80465bbc0134e152994016 (127.6.246.193:40971): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dc089db06e6e48999cb2346a2fc2edcb: Network error: Client connection negotiation failed: client connection to 127.6.246.193:40971: connect: Connection refused (error 111)
I20260925 22:40:03.212081 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 24520
I20260925 22:40:03.216951 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 24158
2026-09-25T22:40:03Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2513 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-25T22:40:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:40:03Z Disabled control of system clock
I20260925 22:40:03.247409 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790375752797504-7131-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790375752797504-7131-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790375752797504-7131-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:42673
--webserver_interface=127.6.246.254
--webserver_port=0
--builtin_ntp_servers=127.6.246.212:37745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.246.254:42673
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 22:40:03.359578 24698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:40:03.359861 24698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:40:03.359896 24698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:40:03.363972 24698 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:40:03.364034 24698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:40:03.364050 24698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:40:03.364066 24698 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:40:03.364089 24698 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:40:03.368783 24698 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:37745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790375752797504-7131-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790375752797504-7131-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.246.254:42673
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:42673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.246.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24698
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790375752797504-7131-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:40:03.369850 24698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:40:03.371170 24698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:40:03.379774 24703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:03.379774 24706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:03.379812 24704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:40:03.380118 24698 server_base.cc:1034] running on GCE node
I20260925 22:40:03.380743 24698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:40:03.381755 24698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:40:03.382925 24698 hybrid_clock.cc:648] HybridClock initialized: now 1790376003382899 us; error 38 us; skew 500 ppm
I20260925 22:40:03.384749 24698 webserver.cc:460] Webserver started at http://127.6.246.254:39029/ using document root <none> and password file <none>
I20260925 22:40:03.385308 24698 fs_manager.cc:359] Metadata directory not provided
I20260925 22:40:03.385370 24698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:40:03.385581 24698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:40:03.387531 24698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790375752797504-7131-0/minicluster-data/master-0/data/instance:
uuid: "b3705af7f90d46d882a44456f8593bbf"
format_stamp: "Formatted at 2026-09-25 22:40:03 on dist-test-slave-wjdr"
server_key: "efee0fc830951c7f8180c5785dfadaa6"
server_key_iv: "9462050b1665d5be01b9c240679e5e49"
server_key_version: "encryptionkey@0"
I20260925 22:40:03.388058 24698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790375752797504-7131-0/minicluster-data/master-0/wal/instance:
uuid: "b3705af7f90d46d882a44456f8593bbf"
format_stamp: "Formatted at 2026-09-25 22:40:03 on dist-test-slave-wjdr"
server_key: "efee0fc830951c7f8180c5785dfadaa6"
server_key_iv: "9462050b1665d5be01b9c240679e5e49"
server_key_version: "encryptionkey@0"
I20260925 22:40:03.391580 24698 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 22:40:03.393967 24712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:03.395174 24698 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 22:40:03.395319 24698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790375752797504-7131-0/minicluster-data/master-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790375752797504-7131-0/minicluster-data/master-0/wal
uuid: "b3705af7f90d46d882a44456f8593bbf"
format_stamp: "Formatted at 2026-09-25 22:40:03 on dist-test-slave-wjdr"
server_key: "efee0fc830951c7f8180c5785dfadaa6"
server_key_iv: "9462050b1665d5be01b9c240679e5e49"
server_key_version: "encryptionkey@0"
I20260925 22:40:03.395430 24698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790375752797504-7131-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790375752797504-7131-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790375752797504-7131-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:40:03.429931 24698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:40:03.430678 24698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:40:03.430887 24698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:40:03.438169 24698 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.254:42673
I20260925 22:40:03.438192 24764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.254:42673 every 8 connection(s)
I20260925 22:40:03.439098 24698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
I20260925 22:40:03.442307 24765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:40:03.443360 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 24698
I20260925 22:40:03.443506 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790375752797504-7131-0/minicluster-data/master-0/wal/instance
I20260925 22:40:03.443782 7131 external_mini_cluster.cc:1428] Setting key c5c425e21abf3655abaaef5277d0f08c
I20260925 22:40:03.448235 24765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3705af7f90d46d882a44456f8593bbf: Bootstrap starting.
I20260925 22:40:03.450928 24765 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3705af7f90d46d882a44456f8593bbf: Neither blocks nor log segments found. Creating new log.
I20260925 22:40:03.451690 24765 log.cc:824] T 00000000000000000000000000000000 P b3705af7f90d46d882a44456f8593bbf: Log is configured to *not* fsync() on all Append() calls
I20260925 22:40:03.452747 24768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:03.445141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37850 (local address 127.6.246.254:42673)
0925 22:40:03.445704 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:03.445718 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:03.445751 (+ 33us) server_negotiation.cc:407] Connection header received
0925 22:40:03.446394 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:03.446409 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:03.446778 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:03.447033 (+ 255us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:03.447407 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:03.448331 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:03.449565 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:03.450000 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:03.450112 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:03.450988 (+ 876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:03.450993 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:03.451004 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:40:03.451438 (+ 434us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:03.451492 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:40:03.451512 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:03.451587 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:03.451745 (+ 158us) server_negotiation.cc:299] Negotiation successful
0925 22:40:03.452003 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":119,"threads_started":1}
I20260925 22:40:03.453672 24765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3705af7f90d46d882a44456f8593bbf: No bootstrap required, opened a new log
I20260925 22:40:03.457212 24765 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3705af7f90d46d882a44456f8593bbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3705af7f90d46d882a44456f8593bbf" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 42673 } }
I20260925 22:40:03.457418 24765 raft_consensus.cc:374] T 00000000000000000000000000000000 P b3705af7f90d46d882a44456f8593bbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:40:03.457490 24765 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3705af7f90d46d882a44456f8593bbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3705af7f90d46d882a44456f8593bbf, State: Initialized, Role: FOLLOWER
I20260925 22:40:03.457893 24765 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3705af7f90d46d882a44456f8593bbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3705af7f90d46d882a44456f8593bbf" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 42673 } }
I20260925 22:40:03.458039 24765 raft_consensus.cc:388] T 00000000000000000000000000000000 P b3705af7f90d46d882a44456f8593bbf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:40:03.458105 24765 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3705af7f90d46d882a44456f8593bbf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:40:03.458243 24765 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3705af7f90d46d882a44456f8593bbf [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:40:03.459148 24765 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3705af7f90d46d882a44456f8593bbf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3705af7f90d46d882a44456f8593bbf" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 42673 } }
I20260925 22:40:03.459471 24765 leader_election.cc:302] T 00000000000000000000000000000000 P b3705af7f90d46d882a44456f8593bbf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3705af7f90d46d882a44456f8593bbf; no voters:
I20260925 22:40:03.459767 24765 leader_election.cc:288] T 00000000000000000000000000000000 P b3705af7f90d46d882a44456f8593bbf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:40:03.459879 24770 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3705af7f90d46d882a44456f8593bbf [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:40:03.460136 24770 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3705af7f90d46d882a44456f8593bbf [term 1 LEADER]: Becoming Leader. State: Replica: b3705af7f90d46d882a44456f8593bbf, State: Running, Role: LEADER
I20260925 22:40:03.460530 24770 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3705af7f90d46d882a44456f8593bbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3705af7f90d46d882a44456f8593bbf" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 42673 } }
I20260925 22:40:03.461036 24765 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3705af7f90d46d882a44456f8593bbf [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:40:03.462383 24772 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3705af7f90d46d882a44456f8593bbf [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3705af7f90d46d882a44456f8593bbf. Latest consensus state: current_term: 1 leader_uuid: "b3705af7f90d46d882a44456f8593bbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3705af7f90d46d882a44456f8593bbf" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 42673 } } }
I20260925 22:40:03.462503 24772 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3705af7f90d46d882a44456f8593bbf [sys.catalog]: This master's current role is: LEADER
I20260925 22:40:03.462440 24771 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3705af7f90d46d882a44456f8593bbf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b3705af7f90d46d882a44456f8593bbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3705af7f90d46d882a44456f8593bbf" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 42673 } } }
I20260925 22:40:03.462555 24771 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3705af7f90d46d882a44456f8593bbf [sys.catalog]: This master's current role is: LEADER
I20260925 22:40:03.462888 24779 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:40:03.465658 24779 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:40:03.470932 24779 catalog_manager.cc:1347] Generated new cluster ID: a9f4ec907c1e47f093c4eefbe3cfd4c5
I20260925 22:40:03.470996 24779 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:40:03.495447 24779 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:40:03.496419 24779 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:40:03.503852 24779 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b3705af7f90d46d882a44456f8593bbf: Generated new TSK 0
I20260925 22:40:03.504580 24779 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:40:03.520236 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790375752797504-7131-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790375752797504-7131-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790375752797504-7131-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.193:0
--local_ip_for_outbound_sockets=127.6.246.193
--webserver_interface=127.6.246.193
--webserver_port=0
--tserver_master_addrs=127.6.246.254:42673
--builtin_ntp_servers=127.6.246.212:37745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 22:40:03.634188 24789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:40:03.634493 24789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:40:03.634534 24789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:40:03.638798 24789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:40:03.638870 24789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:40:03.638928 24789 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.193
I20260925 22:40:03.644500 24789 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:37745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790375752797504-7131-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790375752797504-7131-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.246.193
--webserver_port=0
--tserver_master_addrs=127.6.246.254:42673
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24789
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.193
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790375752797504-7131-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:40:03.645915 24789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:40:03.647364 24789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:40:03.656994 24794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:03.657053 24797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:03.657006 24795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:40:03.657383 24789 server_base.cc:1034] running on GCE node
I20260925 22:40:03.657814 24789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:40:03.658478 24789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:40:03.659660 24789 hybrid_clock.cc:648] HybridClock initialized: now 1790376003659650 us; error 31 us; skew 500 ppm
I20260925 22:40:03.665074 24789 webserver.cc:460] Webserver started at http://127.6.246.193:36195/ using document root <none> and password file <none>
I20260925 22:40:03.665710 24789 fs_manager.cc:359] Metadata directory not provided
I20260925 22:40:03.665812 24789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:40:03.666051 24789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:40:03.668249 24789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790375752797504-7131-0/minicluster-data/ts-0/data/instance:
uuid: "777ac980d71341c29a69e28978d5cc5a"
format_stamp: "Formatted at 2026-09-25 22:40:03 on dist-test-slave-wjdr"
server_key: "c8b1090170b99e604cbc079a427aefe2"
server_key_iv: "102d5eb723feaac16bf7d96a1e2cc240"
server_key_version: "encryptionkey@0"
I20260925 22:40:03.668890 24789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790375752797504-7131-0/minicluster-data/ts-0/wal/instance:
uuid: "777ac980d71341c29a69e28978d5cc5a"
format_stamp: "Formatted at 2026-09-25 22:40:03 on dist-test-slave-wjdr"
server_key: "c8b1090170b99e604cbc079a427aefe2"
server_key_iv: "102d5eb723feaac16bf7d96a1e2cc240"
server_key_version: "encryptionkey@0"
I20260925 22:40:03.673062 24789 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 22:40:03.675994 24803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:03.677340 24789 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:40:03.677476 24789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790375752797504-7131-0/minicluster-data/ts-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790375752797504-7131-0/minicluster-data/ts-0/wal
uuid: "777ac980d71341c29a69e28978d5cc5a"
format_stamp: "Formatted at 2026-09-25 22:40:03 on dist-test-slave-wjdr"
server_key: "c8b1090170b99e604cbc079a427aefe2"
server_key_iv: "102d5eb723feaac16bf7d96a1e2cc240"
server_key_version: "encryptionkey@0"
I20260925 22:40:03.677603 24789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790375752797504-7131-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790375752797504-7131-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790375752797504-7131-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:40:03.699090 24789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:40:03.699836 24789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:40:03.700085 24789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:40:03.700757 24789 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:40:03.701714 24789 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:40:03.701785 24789 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:03.701860 24789 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:40:03.701901 24789 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:03.711289 24789 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.193:42823
I20260925 22:40:03.711313 24916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.193:42823 every 8 connection(s)
I20260925 22:40:03.712246 24789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
I20260925 22:40:03.717197 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 24789
I20260925 22:40:03.717353 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790375752797504-7131-0/minicluster-data/ts-0/wal/instance
I20260925 22:40:03.717622 7131 external_mini_cluster.cc:1428] Setting key e29b232b5a93b44a66962db06850c5c8
I20260925 22:40:03.719520 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790375752797504-7131-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790375752797504-7131-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790375752797504-7131-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.194:0
--local_ip_for_outbound_sockets=127.6.246.194
--webserver_interface=127.6.246.194
--webserver_port=0
--tserver_master_addrs=127.6.246.254:42673
--builtin_ntp_servers=127.6.246.212:37745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 22:40:03.721791 24768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:03.714157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.193:40353 (local address 127.6.246.254:42673)
0925 22:40:03.714334 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:03.714338 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:03.715352 (+ 1014us) server_negotiation.cc:407] Connection header received
0925 22:40:03.716865 (+ 1513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:03.716868 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:03.716932 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:03.717009 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:03.718000 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:03.718556 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:03.719735 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:03.719961 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:03.720018 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:03.721281 (+ 1263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:03.721285 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:03.721288 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:40:03.721300 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:03.721319 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:40:03.721335 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:03.721410 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:03.721570 (+ 160us) server_negotiation.cc:299] Negotiation successful
0925 22:40:03.721640 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260925 22:40:03.722640 24919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:03.714553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:42673 (local address 127.6.246.193:40353)
0925 22:40:03.715197 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:03.715236 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:03.716627 (+ 1391us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:03.717180 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:03.717186 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:03.717273 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:40:03.717860 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:03.717876 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:03.718703 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:03.718706 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:03.719572 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:03.719581 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:03.720171 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:03.720173 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:03.720322 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:03.721016 (+ 694us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:40:03.721018 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:40:03.721057 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:40:03.721120 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:40:03.721427 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:40:03.721442 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:40:03.721447 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 22:40:03.721570 (+ 123us) client_negotiation.cc:240] Negotiation successful
0925 22:40:03.721854 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":114,"threads_started":1}
I20260925 22:40:03.723891 24917 heartbeater.cc:347] Connected to a master server at 127.6.246.254:42673
I20260925 22:40:03.724146 24917 heartbeater.cc:464] Registering TS with master...
I20260925 22:40:03.724726 24917 heartbeater.cc:511] Master 127.6.246.254:42673 requested a full tablet report, sending...
I20260925 22:40:03.726415 24729 ts_manager.cc:194] Registered new tserver with Master: 777ac980d71341c29a69e28978d5cc5a (127.6.246.193:42823)
I20260925 22:40:03.728178 24729 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.246.193:40353
W20260925 22:40:03.835742 24920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:40:03.836050 24920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:40:03.836091 24920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:40:03.840312 24920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:40:03.840392 24920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:40:03.840469 24920 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.194
I20260925 22:40:03.845582 24920 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:37745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790375752797504-7131-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790375752797504-7131-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.246.194
--webserver_port=0
--tserver_master_addrs=127.6.246.254:42673
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24920
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.194
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790375752797504-7131-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:40:03.846664 24920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:40:03.848026 24920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:40:03.858414 24925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:03.858414 24928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:03.858484 24926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:40:03.858603 24920 server_base.cc:1034] running on GCE node
I20260925 22:40:03.859226 24920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:40:03.859910 24920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:40:03.861085 24920 hybrid_clock.cc:648] HybridClock initialized: now 1790376003861072 us; error 27 us; skew 500 ppm
I20260925 22:40:03.863106 24920 webserver.cc:460] Webserver started at http://127.6.246.194:45161/ using document root <none> and password file <none>
I20260925 22:40:03.863755 24920 fs_manager.cc:359] Metadata directory not provided
I20260925 22:40:03.863860 24920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:40:03.864102 24920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:40:03.866309 24920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790375752797504-7131-0/minicluster-data/ts-1/data/instance:
uuid: "ee721d057abc4819884edab8b46308f2"
format_stamp: "Formatted at 2026-09-25 22:40:03 on dist-test-slave-wjdr"
server_key: "a415e3ab18261f74a7636877d5cd7781"
server_key_iv: "ea32c4b2f71cd94429dd0ca58b5ef837"
server_key_version: "encryptionkey@0"
I20260925 22:40:03.866928 24920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790375752797504-7131-0/minicluster-data/ts-1/wal/instance:
uuid: "ee721d057abc4819884edab8b46308f2"
format_stamp: "Formatted at 2026-09-25 22:40:03 on dist-test-slave-wjdr"
server_key: "a415e3ab18261f74a7636877d5cd7781"
server_key_iv: "ea32c4b2f71cd94429dd0ca58b5ef837"
server_key_version: "encryptionkey@0"
I20260925 22:40:03.870813 24920 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260925 22:40:03.873502 24934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:03.874732 24920 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:40:03.874884 24920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790375752797504-7131-0/minicluster-data/ts-1/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790375752797504-7131-0/minicluster-data/ts-1/wal
uuid: "ee721d057abc4819884edab8b46308f2"
format_stamp: "Formatted at 2026-09-25 22:40:03 on dist-test-slave-wjdr"
server_key: "a415e3ab18261f74a7636877d5cd7781"
server_key_iv: "ea32c4b2f71cd94429dd0ca58b5ef837"
server_key_version: "encryptionkey@0"
I20260925 22:40:03.875008 24920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790375752797504-7131-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790375752797504-7131-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790375752797504-7131-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:40:03.926002 24920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:40:03.926744 24920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:40:03.926941 24920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:40:03.927531 24920 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:40:03.928519 24920 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:40:03.928589 24920 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:03.928663 24920 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:40:03.928695 24920 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:03.937907 24920 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.194:33535
I20260925 22:40:03.937934 25047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.194:33535 every 8 connection(s)
I20260925 22:40:03.938911 24920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
I20260925 22:40:03.947026 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 24920
I20260925 22:40:03.947168 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790375752797504-7131-0/minicluster-data/ts-1/wal/instance
I20260925 22:40:03.947515 7131 external_mini_cluster.cc:1428] Setting key 8e3fc981320c355e8d49425dffe75dab
I20260925 22:40:03.947868 24768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:03.940692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.194:48535 (local address 127.6.246.254:42673)
0925 22:40:03.940824 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:03.940828 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:03.941777 (+ 949us) server_negotiation.cc:407] Connection header received
0925 22:40:03.943119 (+ 1342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:03.943122 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:03.943193 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:03.943290 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:03.944197 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:03.944756 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:03.945906 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:03.946121 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:03.946213 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:03.947356 (+ 1143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:03.947361 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:03.947364 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:40:03.947382 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:03.947408 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:40:03.947418 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:03.947492 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:03.947648 (+ 156us) server_negotiation.cc:299] Negotiation successful
0925 22:40:03.947718 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 22:40:03.948786 25050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:03.941002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:42673 (local address 127.6.246.194:48535)
0925 22:40:03.941632 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:03.941671 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:03.942909 (+ 1238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:03.943429 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:03.943435 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:03.943513 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:40:03.943990 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:03.944002 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:03.944891 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:03.944895 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:03.945754 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:03.945763 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:03.946259 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:03.946262 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:03.946400 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:03.947054 (+ 654us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:40:03.947056 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:40:03.947093 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:40:03.947180 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:40:03.947499 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:40:03.947506 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:40:03.947510 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 22:40:03.947633 (+ 123us) client_negotiation.cc:240] Negotiation successful
0925 22:40:03.947941 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":116,"threads_started":1}
I20260925 22:40:03.949715 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790375752797504-7131-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790375752797504-7131-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790375752797504-7131-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.195:0
--local_ip_for_outbound_sockets=127.6.246.195
--webserver_interface=127.6.246.195
--webserver_port=0
--tserver_master_addrs=127.6.246.254:42673
--builtin_ntp_servers=127.6.246.212:37745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 22:40:03.949988 25048 heartbeater.cc:347] Connected to a master server at 127.6.246.254:42673
I20260925 22:40:03.950239 25048 heartbeater.cc:464] Registering TS with master...
I20260925 22:40:03.950763 25048 heartbeater.cc:511] Master 127.6.246.254:42673 requested a full tablet report, sending...
I20260925 22:40:03.951946 24729 ts_manager.cc:194] Registered new tserver with Master: ee721d057abc4819884edab8b46308f2 (127.6.246.194:33535)
I20260925 22:40:03.952615 24729 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.246.194:48535
W20260925 22:40:04.067132 25051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:40:04.067399 25051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:40:04.067440 25051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:40:04.071556 25051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:40:04.071625 25051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:40:04.071734 25051 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.195
I20260925 22:40:04.076630 25051 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:37745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790375752797504-7131-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790375752797504-7131-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.246.195
--webserver_port=0
--tserver_master_addrs=127.6.246.254:42673
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25051
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.195
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790375752797504-7131-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:40:04.077764 25051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:40:04.079311 25051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:40:04.089089 25059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:40:04.089334 25051 server_base.cc:1034] running on GCE node
W20260925 22:40:04.089107 25056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:04.089116 25057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:40:04.089884 25051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:40:04.090502 25051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:40:04.091661 25051 hybrid_clock.cc:648] HybridClock initialized: now 1790376004091641 us; error 32 us; skew 500 ppm
I20260925 22:40:04.093470 25051 webserver.cc:460] Webserver started at http://127.6.246.195:35103/ using document root <none> and password file <none>
I20260925 22:40:04.094065 25051 fs_manager.cc:359] Metadata directory not provided
I20260925 22:40:04.094125 25051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:40:04.094383 25051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:40:04.096408 25051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790375752797504-7131-0/minicluster-data/ts-2/data/instance:
uuid: "07fb922e2ee14c0999bd5ead79cb265e"
format_stamp: "Formatted at 2026-09-25 22:40:04 on dist-test-slave-wjdr"
server_key: "065103782ee59eae25c488d849ca7da3"
server_key_iv: "aad1eb6ad4451d9ea47b52806ce981ea"
server_key_version: "encryptionkey@0"
I20260925 22:40:04.097038 25051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790375752797504-7131-0/minicluster-data/ts-2/wal/instance:
uuid: "07fb922e2ee14c0999bd5ead79cb265e"
format_stamp: "Formatted at 2026-09-25 22:40:04 on dist-test-slave-wjdr"
server_key: "065103782ee59eae25c488d849ca7da3"
server_key_iv: "aad1eb6ad4451d9ea47b52806ce981ea"
server_key_version: "encryptionkey@0"
I20260925 22:40:04.101142 25051 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 22:40:04.103726 25065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:04.105160 25051 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:40:04.105280 25051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790375752797504-7131-0/minicluster-data/ts-2/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790375752797504-7131-0/minicluster-data/ts-2/wal
uuid: "07fb922e2ee14c0999bd5ead79cb265e"
format_stamp: "Formatted at 2026-09-25 22:40:04 on dist-test-slave-wjdr"
server_key: "065103782ee59eae25c488d849ca7da3"
server_key_iv: "aad1eb6ad4451d9ea47b52806ce981ea"
server_key_version: "encryptionkey@0"
I20260925 22:40:04.105429 25051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790375752797504-7131-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790375752797504-7131-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790375752797504-7131-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:40:04.121649 25051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:40:04.122386 25051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:40:04.122592 25051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:40:04.123225 25051 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:40:04.124181 25051 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:40:04.124226 25051 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:04.124265 25051 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:40:04.124331 25051 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:04.134284 25051 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.195:35127
I20260925 22:40:04.134306 25178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.195:35127 every 8 connection(s)
I20260925 22:40:04.135264 25051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
I20260925 22:40:04.136888 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 25051
I20260925 22:40:04.137024 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790375752797504-7131-0/minicluster-data/ts-2/wal/instance
I20260925 22:40:04.137365 7131 external_mini_cluster.cc:1428] Setting key 2c7b295204cfb4840feea2f263e05789
I20260925 22:40:04.145599 24768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:04.137277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.195:35961 (local address 127.6.246.254:42673)
0925 22:40:04.137450 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:04.137454 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:04.138956 (+ 1502us) server_negotiation.cc:407] Connection header received
0925 22:40:04.140583 (+ 1627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:04.140586 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:04.140633 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:04.140704 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:04.141602 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:04.142133 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:04.143548 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:04.143926 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:04.144022 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:04.145164 (+ 1142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:04.145167 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:04.145169 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:40:04.145181 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:04.145200 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:40:04.145208 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:04.145277 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:04.145415 (+ 138us) server_negotiation.cc:299] Negotiation successful
0925 22:40:04.145468 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 22:40:04.146476 25181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:04.137765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:42673 (local address 127.6.246.195:35961)
0925 22:40:04.138749 (+ 984us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:04.138810 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:04.140303 (+ 1493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:04.140872 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:04.140879 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:04.140965 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:40:04.141474 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:04.141485 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:04.142450 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:04.142453 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:04.143421 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:04.143432 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:04.144080 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:04.144083 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:04.144228 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:04.144917 (+ 689us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:40:04.144919 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:40:04.144959 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:40:04.145025 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:40:04.145293 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:40:04.145299 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:40:04.145303 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 22:40:04.145399 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 22:40:04.145622 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":175,"threads_started":1}
I20260925 22:40:04.147811 25179 heartbeater.cc:347] Connected to a master server at 127.6.246.254:42673
I20260925 22:40:04.148065 25179 heartbeater.cc:464] Registering TS with master...
I20260925 22:40:04.148659 25179 heartbeater.cc:511] Master 127.6.246.254:42673 requested a full tablet report, sending...
I20260925 22:40:04.149744 24729 ts_manager.cc:194] Registered new tserver with Master: 07fb922e2ee14c0999bd5ead79cb265e (127.6.246.195:35127)
I20260925 22:40:04.150471 24729 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.246.195:35961
I20260925 22:40:04.151527 7131 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:40:04.157473 24768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:04.153588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37866 (local address 127.6.246.254:42673)
0925 22:40:04.153796 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:04.153800 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:04.154023 (+ 223us) server_negotiation.cc:407] Connection header received
0925 22:40:04.154098 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:04.154100 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:04.154139 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:04.154204 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:04.154502 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:04.155011 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:04.155870 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:04.156078 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:04.156156 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:04.157010 (+ 854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:04.157013 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:04.157015 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:40:04.157026 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:04.157044 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:40:04.157051 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:04.157119 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:04.157216 (+ 97us) server_negotiation.cc:299] Negotiation successful
0925 22:40:04.157262 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260925 22:40:04.160384 24729 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37866:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:40:04.163733 24729 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:40:04.178263 25192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:04.173113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.194:33535 (local address 127.0.0.1:56330)
0925 22:40:04.173736 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:04.173750 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:04.173832 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:04.174564 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:04.174567 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:04.174585 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:40:04.174677 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:04.174682 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:04.175828 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:04.175831 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:04.176924 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:04.176933 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:04.177680 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:04.177683 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:04.177788 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:04.177845 (+ 57us) client_negotiation.cc:769] Sending connection context
0925 22:40:04.177987 (+ 142us) client_negotiation.cc:240] Negotiation successful
0925 22:40:04.178105 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"mutex_wait_us":74,"thread_start_us":98,"threads_started":1}
I20260925 22:40:04.178268 25193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:04.173271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56330 (local address 127.6.246.194:33535)
0925 22:40:04.174129 (+ 858us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:04.174142 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:04.174164 (+ 22us) server_negotiation.cc:407] Connection header received
0925 22:40:04.174244 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:04.174248 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:04.174416 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:04.174568 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:40:04.174806 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:04.175708 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:04.177039 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:04.177552 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:04.177679 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:04.177847 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:04.178018 (+ 171us) server_negotiation.cc:299] Negotiation successful
0925 22:40:04.178110 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":755,"thread_start_us":141,"threads_started":1}
I20260925 22:40:04.178325 25191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:04.172519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36226 (local address 127.6.246.195:35127)
0925 22:40:04.172928 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:04.172940 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:04.172966 (+ 26us) server_negotiation.cc:407] Connection header received
0925 22:40:04.173068 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:04.173073 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:04.173296 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:04.173454 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:40:04.174019 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:04.175008 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:04.176587 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:04.177411 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:04.177490 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:04.177671 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:04.178039 (+ 368us) server_negotiation.cc:299] Negotiation successful
0925 22:40:04.178159 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":166,"threads_started":1}
I20260925 22:40:04.178690 25190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:04.172344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.195:35127 (local address 127.0.0.1:36226)
0925 22:40:04.172726 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:04.172778 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:04.172910 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:04.173619 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:04.173624 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:04.173649 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:40:04.173868 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:04.173878 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:04.175119 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:04.175122 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:04.176433 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:04.176440 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:04.177780 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:04.177786 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:04.177888 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:04.177910 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 22:40:04.178476 (+ 566us) client_negotiation.cc:240] Negotiation successful
0925 22:40:04.178561 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":109,"threads_started":1}
I20260925 22:40:04.179203 25194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:04.173442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.193:42823 (local address 127.0.0.1:36994)
0925 22:40:04.174023 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:04.174036 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:04.174110 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:04.174777 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:04.174780 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:04.174796 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:40:04.174891 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:04.174897 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:04.176320 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:04.176323 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:04.177731 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:04.177739 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:04.178823 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:04.178830 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:04.178923 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:04.178944 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 22:40:04.179004 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 22:40:04.179076 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"mutex_wait_us":30,"thread_start_us":83,"threads_started":1}
I20260925 22:40:04.179379 25195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:04.173622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36994 (local address 127.6.246.193:42823)
0925 22:40:04.174362 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:04.174372 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:04.174393 (+ 21us) server_negotiation.cc:407] Connection header received
0925 22:40:04.174464 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:04.174471 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:04.174630 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:04.174766 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:40:04.175019 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:04.176181 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:04.177859 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:04.178689 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:04.178804 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:04.178992 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:04.179038 (+ 46us) server_negotiation.cc:299] Negotiation successful
0925 22:40:04.179138 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":137,"threads_started":1}
I20260925 22:40:04.181905 24982 tablet_service.cc:1467] Processing CreateTablet for tablet fced297a582c472c90d02c1e4cece905 (DEFAULT_TABLE table=test-table [id=d55d926792b942f88df3469bc17b5150]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:40:04.181947 25113 tablet_service.cc:1467] Processing CreateTablet for tablet fced297a582c472c90d02c1e4cece905 (DEFAULT_TABLE table=test-table [id=d55d926792b942f88df3469bc17b5150]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:40:04.182149 24851 tablet_service.cc:1467] Processing CreateTablet for tablet fced297a582c472c90d02c1e4cece905 (DEFAULT_TABLE table=test-table [id=d55d926792b942f88df3469bc17b5150]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:40:04.182938 24982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fced297a582c472c90d02c1e4cece905. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:40:04.182971 24851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fced297a582c472c90d02c1e4cece905. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:40:04.182971 25113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fced297a582c472c90d02c1e4cece905. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:40:04.189162 25196 tablet_bootstrap.cc:492] T fced297a582c472c90d02c1e4cece905 P ee721d057abc4819884edab8b46308f2: Bootstrap starting.
I20260925 22:40:04.190833 25197 tablet_bootstrap.cc:492] T fced297a582c472c90d02c1e4cece905 P 777ac980d71341c29a69e28978d5cc5a: Bootstrap starting.
I20260925 22:40:04.191581 25196 tablet_bootstrap.cc:654] T fced297a582c472c90d02c1e4cece905 P ee721d057abc4819884edab8b46308f2: Neither blocks nor log segments found. Creating new log.
I20260925 22:40:04.192327 25196 log.cc:824] T fced297a582c472c90d02c1e4cece905 P ee721d057abc4819884edab8b46308f2: Log is configured to *not* fsync() on all Append() calls
I20260925 22:40:04.193157 25197 tablet_bootstrap.cc:654] T fced297a582c472c90d02c1e4cece905 P 777ac980d71341c29a69e28978d5cc5a: Neither blocks nor log segments found. Creating new log.
I20260925 22:40:04.193183 25198 tablet_bootstrap.cc:492] T fced297a582c472c90d02c1e4cece905 P 07fb922e2ee14c0999bd5ead79cb265e: Bootstrap starting.
I20260925 22:40:04.194128 25197 log.cc:824] T fced297a582c472c90d02c1e4cece905 P 777ac980d71341c29a69e28978d5cc5a: Log is configured to *not* fsync() on all Append() calls
I20260925 22:40:04.194558 25196 tablet_bootstrap.cc:492] T fced297a582c472c90d02c1e4cece905 P ee721d057abc4819884edab8b46308f2: No bootstrap required, opened a new log
I20260925 22:40:04.194768 25196 ts_tablet_manager.cc:1397] T fced297a582c472c90d02c1e4cece905 P ee721d057abc4819884edab8b46308f2: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 22:40:04.195360 25198 tablet_bootstrap.cc:654] T fced297a582c472c90d02c1e4cece905 P 07fb922e2ee14c0999bd5ead79cb265e: Neither blocks nor log segments found. Creating new log.
I20260925 22:40:04.195996 25197 tablet_bootstrap.cc:492] T fced297a582c472c90d02c1e4cece905 P 777ac980d71341c29a69e28978d5cc5a: No bootstrap required, opened a new log
I20260925 22:40:04.196121 25198 log.cc:824] T fced297a582c472c90d02c1e4cece905 P 07fb922e2ee14c0999bd5ead79cb265e: Log is configured to *not* fsync() on all Append() calls
I20260925 22:40:04.196233 25197 ts_tablet_manager.cc:1397] T fced297a582c472c90d02c1e4cece905 P 777ac980d71341c29a69e28978d5cc5a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 22:40:04.198150 25198 tablet_bootstrap.cc:492] T fced297a582c472c90d02c1e4cece905 P 07fb922e2ee14c0999bd5ead79cb265e: No bootstrap required, opened a new log
I20260925 22:40:04.198349 25198 ts_tablet_manager.cc:1397] T fced297a582c472c90d02c1e4cece905 P 07fb922e2ee14c0999bd5ead79cb265e: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260925 22:40:04.199159 25196 raft_consensus.cc:348] T fced297a582c472c90d02c1e4cece905 P ee721d057abc4819884edab8b46308f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07fb922e2ee14c0999bd5ead79cb265e" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 35127 } } peers { permanent_uuid: "ee721d057abc4819884edab8b46308f2" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 33535 } } peers { permanent_uuid: "777ac980d71341c29a69e28978d5cc5a" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 42823 } }
I20260925 22:40:04.199445 25196 raft_consensus.cc:374] T fced297a582c472c90d02c1e4cece905 P ee721d057abc4819884edab8b46308f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:40:04.199533 25196 raft_consensus.cc:729] T fced297a582c472c90d02c1e4cece905 P ee721d057abc4819884edab8b46308f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee721d057abc4819884edab8b46308f2, State: Initialized, Role: FOLLOWER
I20260925 22:40:04.200047 25196 consensus_queue.cc:260] T fced297a582c472c90d02c1e4cece905 P ee721d057abc4819884edab8b46308f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07fb922e2ee14c0999bd5ead79cb265e" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 35127 } } peers { permanent_uuid: "ee721d057abc4819884edab8b46308f2" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 33535 } } peers { permanent_uuid: "777ac980d71341c29a69e28978d5cc5a" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 42823 } }
I20260925 22:40:04.200269 25197 raft_consensus.cc:348] T fced297a582c472c90d02c1e4cece905 P 777ac980d71341c29a69e28978d5cc5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07fb922e2ee14c0999bd5ead79cb265e" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 35127 } } peers { permanent_uuid: "ee721d057abc4819884edab8b46308f2" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 33535 } } peers { permanent_uuid: "777ac980d71341c29a69e28978d5cc5a" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 42823 } }
I20260925 22:40:04.200814 25048 heartbeater.cc:503] Master 127.6.246.254:42673 was elected leader, sending a full tablet report...
I20260925 22:40:04.200980 25197 raft_consensus.cc:374] T fced297a582c472c90d02c1e4cece905 P 777ac980d71341c29a69e28978d5cc5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:40:04.201049 25197 raft_consensus.cc:729] T fced297a582c472c90d02c1e4cece905 P 777ac980d71341c29a69e28978d5cc5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 777ac980d71341c29a69e28978d5cc5a, State: Initialized, Role: FOLLOWER
I20260925 22:40:04.201134 25196 ts_tablet_manager.cc:1428] T fced297a582c472c90d02c1e4cece905 P ee721d057abc4819884edab8b46308f2: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 22:40:04.201462 25197 consensus_queue.cc:260] T fced297a582c472c90d02c1e4cece905 P 777ac980d71341c29a69e28978d5cc5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07fb922e2ee14c0999bd5ead79cb265e" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 35127 } } peers { permanent_uuid: "ee721d057abc4819884edab8b46308f2" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 33535 } } peers { permanent_uuid: "777ac980d71341c29a69e28978d5cc5a" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 42823 } }
I20260925 22:40:04.202158 24917 heartbeater.cc:503] Master 127.6.246.254:42673 was elected leader, sending a full tablet report...
I20260925 22:40:04.202441 25197 ts_tablet_manager.cc:1428] T fced297a582c472c90d02c1e4cece905 P 777ac980d71341c29a69e28978d5cc5a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 22:40:04.202631 25198 raft_consensus.cc:348] T fced297a582c472c90d02c1e4cece905 P 07fb922e2ee14c0999bd5ead79cb265e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07fb922e2ee14c0999bd5ead79cb265e" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 35127 } } peers { permanent_uuid: "ee721d057abc4819884edab8b46308f2" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 33535 } } peers { permanent_uuid: "777ac980d71341c29a69e28978d5cc5a" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 42823 } }
I20260925 22:40:04.202863 25198 raft_consensus.cc:374] T fced297a582c472c90d02c1e4cece905 P 07fb922e2ee14c0999bd5ead79cb265e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:40:04.202922 25198 raft_consensus.cc:729] T fced297a582c472c90d02c1e4cece905 P 07fb922e2ee14c0999bd5ead79cb265e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07fb922e2ee14c0999bd5ead79cb265e, State: Initialized, Role: FOLLOWER
I20260925 22:40:04.203354 25198 consensus_queue.cc:260] T fced297a582c472c90d02c1e4cece905 P 07fb922e2ee14c0999bd5ead79cb265e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07fb922e2ee14c0999bd5ead79cb265e" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 35127 } } peers { permanent_uuid: "ee721d057abc4819884edab8b46308f2" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 33535 } } peers { permanent_uuid: "777ac980d71341c29a69e28978d5cc5a" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 42823 } }
I20260925 22:40:04.204075 25179 heartbeater.cc:503] Master 127.6.246.254:42673 was elected leader, sending a full tablet report...
I20260925 22:40:04.204370 25198 ts_tablet_manager.cc:1428] T fced297a582c472c90d02c1e4cece905 P 07fb922e2ee14c0999bd5ead79cb265e: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
W20260925 22:40:04.213472 24918 tablet.cc:2369] T fced297a582c472c90d02c1e4cece905 P 777ac980d71341c29a69e28978d5cc5a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:40:04.310958 25204 raft_consensus.cc:482] T fced297a582c472c90d02c1e4cece905 P 07fb922e2ee14c0999bd5ead79cb265e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:40:04.311215 25204 raft_consensus.cc:504] T fced297a582c472c90d02c1e4cece905 P 07fb922e2ee14c0999bd5ead79cb265e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07fb922e2ee14c0999bd5ead79cb265e" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 35127 } } peers { permanent_uuid: "ee721d057abc4819884edab8b46308f2" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 33535 } } peers { permanent_uuid: "777ac980d71341c29a69e28978d5cc5a" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 42823 } }
I20260925 22:40:04.312398 25204 leader_election.cc:288] T fced297a582c472c90d02c1e4cece905 P 07fb922e2ee14c0999bd5ead79cb265e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee721d057abc4819884edab8b46308f2 (127.6.246.194:33535), 777ac980d71341c29a69e28978d5cc5a (127.6.246.193:42823)
I20260925 22:40:04.316398 25193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:04.312793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.195:54013 (local address 127.6.246.194:33535)
0925 22:40:04.312927 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:04.312931 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:04.312946 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:40:04.313031 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:04.313034 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:04.313108 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:04.313235 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:40:04.313521 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:04.314149 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:04.315435 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:04.315831 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:04.315972 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:04.316023 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:04.316219 (+ 196us) server_negotiation.cc:299] Negotiation successful
0925 22:40:04.316292 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 22:40:04.316398 25181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:04.312677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.194:33535 (local address 127.6.246.195:54013)
0925 22:40:04.312815 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:04.312834 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:04.312935 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:04.313241 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:04.313244 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:04.313263 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:40:04.313392 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:04.313399 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:04.314319 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:04.314322 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:04.315301 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:04.315311 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:04.315963 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:04.315967 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:04.316064 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:04.316090 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 22:40:04.316184 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 22:40:04.316262 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":24}
I20260925 22:40:04.316592 25195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:04.313205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.195:56035 (local address 127.6.246.193:42823)
0925 22:40:04.313366 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:04.313370 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:04.313532 (+ 162us) server_negotiation.cc:407] Connection header received
0925 22:40:04.313651 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:04.313654 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:04.313705 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:04.313813 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:40:04.314098 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:04.314577 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:04.315616 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:04.316004 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:04.316142 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:04.316187 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:04.316380 (+ 193us) server_negotiation.cc:299] Negotiation successful
0925 22:40:04.316470 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 22:40:04.316606 25205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:04.313026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.193:42823 (local address 127.6.246.195:56035)
0925 22:40:04.313426 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:04.313438 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:04.313558 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:04.313830 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:04.313833 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:04.313849 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:40:04.313984 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:04.313989 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:04.314695 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:04.314707 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:04.315487 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:04.315495 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:04.316143 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:04.316146 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:04.316244 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:04.316265 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 22:40:04.316364 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 22:40:04.316429 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"mutex_wait_us":1,"thread_start_us":138,"threads_started":1}
I20260925 22:40:04.317183 24871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fced297a582c472c90d02c1e4cece905" candidate_uuid: "07fb922e2ee14c0999bd5ead79cb265e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "777ac980d71341c29a69e28978d5cc5a" is_pre_election: true
I20260925 22:40:04.317236 25002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fced297a582c472c90d02c1e4cece905" candidate_uuid: "07fb922e2ee14c0999bd5ead79cb265e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee721d057abc4819884edab8b46308f2" is_pre_election: true
I20260925 22:40:04.317502 24871 raft_consensus.cc:2454] T fced297a582c472c90d02c1e4cece905 P 777ac980d71341c29a69e28978d5cc5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07fb922e2ee14c0999bd5ead79cb265e in term 0.
I20260925 22:40:04.317523 25002 raft_consensus.cc:2454] T fced297a582c472c90d02c1e4cece905 P ee721d057abc4819884edab8b46308f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07fb922e2ee14c0999bd5ead79cb265e in term 0.
I20260925 22:40:04.318013 25068 leader_election.cc:302] T fced297a582c472c90d02c1e4cece905 P 07fb922e2ee14c0999bd5ead79cb265e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07fb922e2ee14c0999bd5ead79cb265e, 777ac980d71341c29a69e28978d5cc5a; no voters:
I20260925 22:40:04.318300 25204 raft_consensus.cc:2781] T fced297a582c472c90d02c1e4cece905 P 07fb922e2ee14c0999bd5ead79cb265e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:40:04.318385 25204 raft_consensus.cc:482] T fced297a582c472c90d02c1e4cece905 P 07fb922e2ee14c0999bd5ead79cb265e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:40:04.318428 25204 raft_consensus.cc:3037] T fced297a582c472c90d02c1e4cece905 P 07fb922e2ee14c0999bd5ead79cb265e [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:40:04.319428 25204 raft_consensus.cc:504] T fced297a582c472c90d02c1e4cece905 P 07fb922e2ee14c0999bd5ead79cb265e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07fb922e2ee14c0999bd5ead79cb265e" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 35127 } } peers { permanent_uuid: "ee721d057abc4819884edab8b46308f2" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 33535 } } peers { permanent_uuid: "777ac980d71341c29a69e28978d5cc5a" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 42823 } }
I20260925 22:40:04.319763 25204 leader_election.cc:288] T fced297a582c472c90d02c1e4cece905 P 07fb922e2ee14c0999bd5ead79cb265e [CANDIDATE]: Term 1 election: Requested vote from peers ee721d057abc4819884edab8b46308f2 (127.6.246.194:33535), 777ac980d71341c29a69e28978d5cc5a (127.6.246.193:42823)
I20260925 22:40:04.320165 25002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fced297a582c472c90d02c1e4cece905" candidate_uuid: "07fb922e2ee14c0999bd5ead79cb265e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee721d057abc4819884edab8b46308f2"
I20260925 22:40:04.320166 24871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fced297a582c472c90d02c1e4cece905" candidate_uuid: "07fb922e2ee14c0999bd5ead79cb265e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "777ac980d71341c29a69e28978d5cc5a"
I20260925 22:40:04.320267 25002 raft_consensus.cc:3037] T fced297a582c472c90d02c1e4cece905 P ee721d057abc4819884edab8b46308f2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:40:04.320328 24871 raft_consensus.cc:3037] T fced297a582c472c90d02c1e4cece905 P 777ac980d71341c29a69e28978d5cc5a [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:40:04.321357 25002 raft_consensus.cc:2454] T fced297a582c472c90d02c1e4cece905 P ee721d057abc4819884edab8b46308f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07fb922e2ee14c0999bd5ead79cb265e in term 1.
I20260925 22:40:04.321384 24871 raft_consensus.cc:2454] T fced297a582c472c90d02c1e4cece905 P 777ac980d71341c29a69e28978d5cc5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07fb922e2ee14c0999bd5ead79cb265e in term 1.
I20260925 22:40:04.321677 25068 leader_election.cc:302] T fced297a582c472c90d02c1e4cece905 P 07fb922e2ee14c0999bd5ead79cb265e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07fb922e2ee14c0999bd5ead79cb265e, ee721d057abc4819884edab8b46308f2; no voters:
I20260925 22:40:04.321869 25204 raft_consensus.cc:2781] T fced297a582c472c90d02c1e4cece905 P 07fb922e2ee14c0999bd5ead79cb265e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:40:04.322106 25204 raft_consensus.cc:686] T fced297a582c472c90d02c1e4cece905 P 07fb922e2ee14c0999bd5ead79cb265e [term 1 LEADER]: Becoming Leader. State: Replica: 07fb922e2ee14c0999bd5ead79cb265e, State: Running, Role: LEADER
I20260925 22:40:04.322456 25204 consensus_queue.cc:237] T fced297a582c472c90d02c1e4cece905 P 07fb922e2ee14c0999bd5ead79cb265e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07fb922e2ee14c0999bd5ead79cb265e" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 35127 } } peers { permanent_uuid: "ee721d057abc4819884edab8b46308f2" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 33535 } } peers { permanent_uuid: "777ac980d71341c29a69e28978d5cc5a" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 42823 } }
I20260925 22:40:04.325783 24728 catalog_manager.cc:5495] T fced297a582c472c90d02c1e4cece905 P 07fb922e2ee14c0999bd5ead79cb265e reported cstate change: term changed from 0 to 1, leader changed from <none> to 07fb922e2ee14c0999bd5ead79cb265e (127.6.246.195). New cstate: current_term: 1 leader_uuid: "07fb922e2ee14c0999bd5ead79cb265e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07fb922e2ee14c0999bd5ead79cb265e" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 35127 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee721d057abc4819884edab8b46308f2" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 33535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "777ac980d71341c29a69e28978d5cc5a" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 42823 } health_report { overall_health: UNKNOWN } } }
I20260925 22:40:04.372082 25191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:04.368425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36230 (local address 127.6.246.195:35127)
0925 22:40:04.368582 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:04.368587 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:04.368602 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:40:04.368745 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:04.368747 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:04.368805 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:04.368925 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:40:04.369182 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:04.369702 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:04.370568 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:04.370790 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:04.370909 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:04.371225 (+ 316us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:40:04.371650 (+ 425us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:40:04.371792 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:04.371859 (+ 67us) server_negotiation.cc:299] Negotiation successful
0925 22:40:04.371911 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 22:40:04.378250 24871 raft_consensus.cc:1261] T fced297a582c472c90d02c1e4cece905 P 777ac980d71341c29a69e28978d5cc5a [term 1 FOLLOWER]: Refusing update from remote peer 07fb922e2ee14c0999bd5ead79cb265e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:40:04.378250 25002 raft_consensus.cc:1261] T fced297a582c472c90d02c1e4cece905 P ee721d057abc4819884edab8b46308f2 [term 1 FOLLOWER]: Refusing update from remote peer 07fb922e2ee14c0999bd5ead79cb265e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:40:04.379026 25206 consensus_queue.cc:1035] T fced297a582c472c90d02c1e4cece905 P 07fb922e2ee14c0999bd5ead79cb265e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee721d057abc4819884edab8b46308f2" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 33535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:40:04.379230 25204 consensus_queue.cc:1035] T fced297a582c472c90d02c1e4cece905 P 07fb922e2ee14c0999bd5ead79cb265e [LEADER]: Connected to new peer: Peer: permanent_uuid: "777ac980d71341c29a69e28978d5cc5a" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 42823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 22:40:04.386236 25180 tablet.cc:2369] T fced297a582c472c90d02c1e4cece905 P 07fb922e2ee14c0999bd5ead79cb265e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:40:04.387240 25210 mvcc.cc:205] Tried to move back new op lower bound from 7333380113924800512 to 7333380113707638784. Current Snapshot: MvccSnapshot[applied={T|T < 7333380113924800512}]
I20260925 22:40:04.387861 25211 mvcc.cc:205] Tried to move back new op lower bound from 7333380113924800512 to 7333380113707638784. Current Snapshot: MvccSnapshot[applied={T|T < 7333380113924800512}]
I20260925 22:40:04.396812 24728 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37866:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:40:04.397003 24728 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37866:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:40:04.399417 24728 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b3705af7f90d46d882a44456f8593bbf: Sending DeleteTablet for 3 replicas of tablet fced297a582c472c90d02c1e4cece905
I20260925 22:40:04.400189 25113 tablet_service.cc:1514] Processing DeleteTablet for tablet fced297a582c472c90d02c1e4cece905 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:40:04 UTC) from {username='slave'} at 127.0.0.1:36226
I20260925 22:40:04.400415 24851 tablet_service.cc:1514] Processing DeleteTablet for tablet fced297a582c472c90d02c1e4cece905 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:40:04 UTC) from {username='slave'} at 127.0.0.1:36994
I20260925 22:40:04.400749 24982 tablet_service.cc:1514] Processing DeleteTablet for tablet fced297a582c472c90d02c1e4cece905 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:40:04 UTC) from {username='slave'} at 127.0.0.1:56330
I20260925 22:40:04.400949 25220 tablet_replica.cc:330] T fced297a582c472c90d02c1e4cece905 P 777ac980d71341c29a69e28978d5cc5a: stopping tablet replica
I20260925 22:40:04.401221 25221 tablet_replica.cc:330] T fced297a582c472c90d02c1e4cece905 P ee721d057abc4819884edab8b46308f2: stopping tablet replica
I20260925 22:40:04.401273 25220 raft_consensus.cc:2229] T fced297a582c472c90d02c1e4cece905 P 777ac980d71341c29a69e28978d5cc5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:40:04.401561 25219 tablet_replica.cc:330] T fced297a582c472c90d02c1e4cece905 P 07fb922e2ee14c0999bd5ead79cb265e: stopping tablet replica
I20260925 22:40:04.401687 25220 raft_consensus.cc:2258] T fced297a582c472c90d02c1e4cece905 P 777ac980d71341c29a69e28978d5cc5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:40:04.401867 25219 raft_consensus.cc:2229] T fced297a582c472c90d02c1e4cece905 P 07fb922e2ee14c0999bd5ead79cb265e [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:40:04.401921 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 24789
I20260925 22:40:04.402760 25219 raft_consensus.cc:2258] T fced297a582c472c90d02c1e4cece905 P 07fb922e2ee14c0999bd5ead79cb265e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:40:04.405959 25219 ts_tablet_manager.cc:1905] T fced297a582c472c90d02c1e4cece905 P 07fb922e2ee14c0999bd5ead79cb265e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:40:04.409511 25219 ts_tablet_manager.cc:1918] T fced297a582c472c90d02c1e4cece905 P 07fb922e2ee14c0999bd5ead79cb265e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:40:04.409612 25219 log.cc:1196] T fced297a582c472c90d02c1e4cece905 P 07fb922e2ee14c0999bd5ead79cb265e: Deleting WAL directory at /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790375752797504-7131-0/minicluster-data/ts-2/wal/wals/fced297a582c472c90d02c1e4cece905
I20260925 22:40:04.409932 25219 ts_tablet_manager.cc:1939] T fced297a582c472c90d02c1e4cece905 P 07fb922e2ee14c0999bd5ead79cb265e: Deleting consensus metadata
I20260925 22:40:04.410902 25221 raft_consensus.cc:2229] T fced297a582c472c90d02c1e4cece905 P ee721d057abc4819884edab8b46308f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:40:04.411135 24715 catalog_manager.cc:4841] TS 07fb922e2ee14c0999bd5ead79cb265e (127.6.246.195:35127): tablet fced297a582c472c90d02c1e4cece905 (table test-table [id=d55d926792b942f88df3469bc17b5150]) successfully deleted
I20260925 22:40:04.411228 25221 raft_consensus.cc:2258] T fced297a582c472c90d02c1e4cece905 P ee721d057abc4819884edab8b46308f2 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260925 22:40:04.411705 24715 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.246.193:42823 (error 108)
I20260925 22:40:04.411865 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 24920
I20260925 22:40:04.413440 25190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:04.413074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.193:42823 (local address 127.0.0.1:37006)
0925 22:40:04.413264 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:04.413338 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.193:42823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
W20260925 22:40:04.413610 24715 catalog_manager.cc:4584] TS 777ac980d71341c29a69e28978d5cc5a (127.6.246.193:42823): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fced297a582c472c90d02c1e4cece905: Network error: Client connection negotiation failed: client connection to 127.6.246.193:42823: connect: Connection refused (error 111)
I20260925 22:40:04.418485 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 25051
I20260925 22:40:04.418669 25190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:04.418362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.194:33535 (local address 127.0.0.1:56334)
0925 22:40:04.418509 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:04.418575 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.194:33535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260925 22:40:04.424638 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 24698
2026-09-25T22:40:04Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1206 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:40:04 dist-test-slave-wjdr krb5kdc[25225](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:40:04 dist-test-slave-wjdr krb5kdc[25225](info): set up 2 sockets
Sep 25 22:40:04 dist-test-slave-wjdr krb5kdc[25225](info): commencing operation
krb5kdc: starting...
W20260925 22:40:05.460664 7131 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Sep 25 22:40:05 dist-test-slave-wjdr krb5kdc[25225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:40:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:40:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.246.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab.
Entry for principal kudu/127.6.246.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab.
I20260925 22:40:05.583093 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:33155
--webserver_interface=127.6.246.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab
--principal=kudu/127.6.246.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:40715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.246.254:33155
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:40:05.692513 25238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:40:05.692785 25238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:40:05.692831 25238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:40:05.696844 25238 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:40:05.696916 25238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:40:05.696938 25238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:40:05.696954 25238 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:40:05.696969 25238 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:40:05.702134 25238 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:40715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.246.254:33155
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab
--principal=kudu/127.6.246.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:33155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.246.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25238
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:40:05.703110 25238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:40:05.704046 25238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:40:05.712401 25246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:05.712422 25243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:40:05.712570 25238 server_base.cc:1034] running on GCE node
W20260925 22:40:05.712400 25244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:40:05.713368 25238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:40:05.714465 25238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:40:05.715656 25238 hybrid_clock.cc:648] HybridClock initialized: now 1790376005715638 us; error 31 us; skew 500 ppm
Sep 25 22:40:05 dist-test-slave-wjdr krb5kdc[25225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376005, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:40:05.718400 25238 init.cc:377] Logged in from keytab as kudu/127.6.246.254@KRBTEST.COM (short username kudu)
I20260925 22:40:05.719254 25238 webserver.cc:460] Webserver started at http://127.6.246.254:33811/ using document root <none> and password file <none>
I20260925 22:40:05.719820 25238 fs_manager.cc:359] Metadata directory not provided
I20260925 22:40:05.719911 25238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:40:05.720140 25238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:40:05.722191 25238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/minicluster-data/master-0/data/instance:
uuid: "1d1ceac58a6043c1a093a43d49a95994"
format_stamp: "Formatted at 2026-09-25 22:40:05 on dist-test-slave-wjdr"
server_key: "9fae150a89cb128380eb2ffb84d10c6f"
server_key_iv: "ebb06597ca5a35b47fa448f28cc0e8c5"
server_key_version: "encryptionkey@0"
I20260925 22:40:05.722766 25238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/minicluster-data/master-0/wal/instance:
uuid: "1d1ceac58a6043c1a093a43d49a95994"
format_stamp: "Formatted at 2026-09-25 22:40:05 on dist-test-slave-wjdr"
server_key: "9fae150a89cb128380eb2ffb84d10c6f"
server_key_iv: "ebb06597ca5a35b47fa448f28cc0e8c5"
server_key_version: "encryptionkey@0"
I20260925 22:40:05.726432 25238 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 22:40:05.728870 25253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:05.730057 25238 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:40:05.730221 25238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/minicluster-data/master-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/minicluster-data/master-0/wal
uuid: "1d1ceac58a6043c1a093a43d49a95994"
format_stamp: "Formatted at 2026-09-25 22:40:05 on dist-test-slave-wjdr"
server_key: "9fae150a89cb128380eb2ffb84d10c6f"
server_key_iv: "ebb06597ca5a35b47fa448f28cc0e8c5"
server_key_version: "encryptionkey@0"
I20260925 22:40:05.730348 25238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:40:05.745445 25238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:40:05.748304 25238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:40:05.748528 25238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:40:05.755475 25238 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.254:33155
I20260925 22:40:05.755550 25305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.254:33155 every 8 connection(s)
I20260925 22:40:05.756424 25238 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
I20260925 22:40:05.758925 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 25238
I20260925 22:40:05.759076 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/minicluster-data/master-0/wal/instance
I20260925 22:40:05.759308 25306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:40:05.759387 7131 external_mini_cluster.cc:1428] Setting key b5843f20a3e138a9aac105d1aefb2645
I20260925 22:40:05.766712 25306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d1ceac58a6043c1a093a43d49a95994: Bootstrap starting.
Sep 25 22:40:05 dist-test-slave-wjdr krb5kdc[25225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
I20260925 22:40:05.769197 25306 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d1ceac58a6043c1a093a43d49a95994: Neither blocks nor log segments found. Creating new log.
I20260925 22:40:05.770145 25306 log.cc:824] T 00000000000000000000000000000000 P 1d1ceac58a6043c1a093a43d49a95994: Log is configured to *not* fsync() on all Append() calls
I20260925 22:40:05.772466 25306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d1ceac58a6043c1a093a43d49a95994: No bootstrap required, opened a new log
I20260925 22:40:05.774654 25309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:05.760851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57476 (local address 127.6.246.254:33155)
0925 22:40:05.761268 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:05.761286 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:05.761327 (+ 41us) server_negotiation.cc:407] Connection header received
0925 22:40:05.762179 (+ 852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:05.762203 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:05.762615 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:05.762988 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:05.763659 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:05.765125 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:05.766045 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:05.766701 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:05.766876 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:05.769532 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:05.769539 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:05.769556 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:05.769588 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:05.771471 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:05.771944 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:05.771950 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:05.771955 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:05.772088 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:05.772401 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:05.772404 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:05.772406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:05.772956 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:05.773149 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:05.773408 (+ 259us) server_negotiation.cc:299] Negotiation successful
0925 22:40:05.773695 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":105,"threads_started":1}
I20260925 22:40:05.776963 25306 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d1ceac58a6043c1a093a43d49a95994 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d1ceac58a6043c1a093a43d49a95994" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 33155 } }
I20260925 22:40:05.777205 25306 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d1ceac58a6043c1a093a43d49a95994 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:40:05.777240 25306 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d1ceac58a6043c1a093a43d49a95994 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d1ceac58a6043c1a093a43d49a95994, State: Initialized, Role: FOLLOWER
I20260925 22:40:05.777643 25306 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d1ceac58a6043c1a093a43d49a95994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d1ceac58a6043c1a093a43d49a95994" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 33155 } }
I20260925 22:40:05.777763 25306 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d1ceac58a6043c1a093a43d49a95994 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:40:05.777809 25306 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d1ceac58a6043c1a093a43d49a95994 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:40:05.777880 25306 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d1ceac58a6043c1a093a43d49a95994 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:40:05.778702 25306 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d1ceac58a6043c1a093a43d49a95994 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d1ceac58a6043c1a093a43d49a95994" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 33155 } }
I20260925 22:40:05.778975 25306 leader_election.cc:302] T 00000000000000000000000000000000 P 1d1ceac58a6043c1a093a43d49a95994 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d1ceac58a6043c1a093a43d49a95994; no voters:
I20260925 22:40:05.779210 25306 leader_election.cc:288] T 00000000000000000000000000000000 P 1d1ceac58a6043c1a093a43d49a95994 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:40:05.779340 25311 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d1ceac58a6043c1a093a43d49a95994 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:40:05.779609 25311 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d1ceac58a6043c1a093a43d49a95994 [term 1 LEADER]: Becoming Leader. State: Replica: 1d1ceac58a6043c1a093a43d49a95994, State: Running, Role: LEADER
I20260925 22:40:05.779969 25311 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d1ceac58a6043c1a093a43d49a95994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d1ceac58a6043c1a093a43d49a95994" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 33155 } }
I20260925 22:40:05.780503 25306 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d1ceac58a6043c1a093a43d49a95994 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:40:05.781695 25312 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d1ceac58a6043c1a093a43d49a95994 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d1ceac58a6043c1a093a43d49a95994. Latest consensus state: current_term: 1 leader_uuid: "1d1ceac58a6043c1a093a43d49a95994" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d1ceac58a6043c1a093a43d49a95994" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 33155 } } }
I20260925 22:40:05.781700 25313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d1ceac58a6043c1a093a43d49a95994 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d1ceac58a6043c1a093a43d49a95994" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d1ceac58a6043c1a093a43d49a95994" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 33155 } } }
I20260925 22:40:05.781904 25313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d1ceac58a6043c1a093a43d49a95994 [sys.catalog]: This master's current role is: LEADER
I20260925 22:40:05.782203 25312 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d1ceac58a6043c1a093a43d49a95994 [sys.catalog]: This master's current role is: LEADER
I20260925 22:40:05.782286 25321 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:40:05.785609 25321 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:40:05.790612 25321 catalog_manager.cc:1347] Generated new cluster ID: 8c4b14a14f104675b36e2739973ea0e6
I20260925 22:40:05.790674 25321 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:40:05.798542 25321 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:40:05.799482 25321 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:40:05.806314 25321 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d1ceac58a6043c1a093a43d49a95994: Generated new TSK 0
I20260925 22:40:05.807041 25321 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.246.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.193.keytab.
Entry for principal kudu/127.6.246.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.193.keytab.
I20260925 22:40:05.844183 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.193:0
--local_ip_for_outbound_sockets=127.6.246.193
--webserver_interface=127.6.246.193
--webserver_port=0
--tserver_master_addrs=127.6.246.254:33155
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.193.keytab
--principal=kudu/127.6.246.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:40715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:40:05.956851 25332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:40:05.957082 25332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:40:05.957147 25332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:40:05.961251 25332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:40:05.961308 25332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:40:05.961364 25332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.193
I20260925 22:40:05.966460 25332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:40715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.193.keytab
--principal=kudu/127.6.246.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.246.193
--webserver_port=0
--tserver_master_addrs=127.6.246.254:33155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25332
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.193
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:40:05.967414 25332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:40:05.968345 25332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:40:05.977923 25340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:05.977932 25337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:40:05.978025 25332 server_base.cc:1034] running on GCE node
W20260925 22:40:05.977936 25338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:40:05.978549 25332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:40:05.979152 25332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:40:05.980307 25332 hybrid_clock.cc:648] HybridClock initialized: now 1790376005980290 us; error 33 us; skew 500 ppm
Sep 25 22:40:05 dist-test-slave-wjdr krb5kdc[25225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376005, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:40:05.983361 25332 init.cc:377] Logged in from keytab as kudu/127.6.246.193@KRBTEST.COM (short username kudu)
I20260925 22:40:05.984205 25332 webserver.cc:460] Webserver started at http://127.6.246.193:44319/ using document root <none> and password file <none>
I20260925 22:40:05.984815 25332 fs_manager.cc:359] Metadata directory not provided
I20260925 22:40:05.984920 25332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:40:05.985150 25332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:40:05.987201 25332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/minicluster-data/ts-0/data/instance:
uuid: "d44e1943717546bc866473b91e63a64f"
format_stamp: "Formatted at 2026-09-25 22:40:05 on dist-test-slave-wjdr"
server_key: "50c220b077093cde854295f684760f54"
server_key_iv: "39ede2750a5517f7dd2f5afbf33d76e6"
server_key_version: "encryptionkey@0"
I20260925 22:40:05.987782 25332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/minicluster-data/ts-0/wal/instance:
uuid: "d44e1943717546bc866473b91e63a64f"
format_stamp: "Formatted at 2026-09-25 22:40:05 on dist-test-slave-wjdr"
server_key: "50c220b077093cde854295f684760f54"
server_key_iv: "39ede2750a5517f7dd2f5afbf33d76e6"
server_key_version: "encryptionkey@0"
I20260925 22:40:05.991353 25332 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 22:40:05.993685 25347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:05.994887 25332 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 22:40:05.995031 25332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/minicluster-data/ts-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/minicluster-data/ts-0/wal
uuid: "d44e1943717546bc866473b91e63a64f"
format_stamp: "Formatted at 2026-09-25 22:40:05 on dist-test-slave-wjdr"
server_key: "50c220b077093cde854295f684760f54"
server_key_iv: "39ede2750a5517f7dd2f5afbf33d76e6"
server_key_version: "encryptionkey@0"
I20260925 22:40:05.995141 25332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:40:06.020072 25332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:40:06.023558 25332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:40:06.023785 25332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:40:06.024418 25332 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:40:06.025275 25332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:40:06.025362 25332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:06.025455 25332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:40:06.025509 25332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:06.035347 25332 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.193:35911
I20260925 22:40:06.035367 25460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.193:35911 every 8 connection(s)
I20260925 22:40:06.036226 25332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
I20260925 22:40:06.041368 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 25332
I20260925 22:40:06.041517 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/minicluster-data/ts-0/wal/instance
I20260925 22:40:06.041805 7131 external_mini_cluster.cc:1428] Setting key 7ae80a9a5d2316f4af68bfdcae5c257e
Sep 25 22:40:06 dist-test-slave-wjdr krb5kdc[25225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376005, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.193@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.246.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.194@KRBTEST.COM" created.
I20260925 22:40:06.060081 25309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:06.038106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.193:43691 (local address 127.6.246.254:33155)
0925 22:40:06.038258 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:06.038262 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:06.039057 (+ 795us) server_negotiation.cc:407] Connection header received
0925 22:40:06.039946 (+ 889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:06.039949 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:06.040005 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:06.040120 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:06.041321 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:06.042017 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:06.043183 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:06.043432 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:06.043555 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:06.054919 (+ 11364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:06.054927 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:06.054931 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:06.054969 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:06.057426 (+ 2457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:06.058175 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:06.058179 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:06.058180 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:06.058277 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:06.058728 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:06.058732 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:06.058734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:06.058956 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:06.059078 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:06.059742 (+ 664us) server_negotiation.cc:299] Negotiation successful
0925 22:40:06.059900 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 22:40:06.061234 25463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:06.038469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:33155 (local address 127.6.246.193:43691)
0925 22:40:06.038905 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:06.038946 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:06.039743 (+ 797us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:06.040283 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:06.040289 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:06.040657 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:40:06.041156 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:06.041175 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:06.042147 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:06.042150 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:06.043039 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:06.043049 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:06.043594 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:06.043598 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:06.043827 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:06.044602 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:40:06.044625 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:40:06.054616 (+ 9991us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:40:06.057617 (+ 3001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:06.057625 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:06.057646 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:06.058056 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:06.058389 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:06.058395 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:06.058400 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:06.058609 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:06.059074 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:40:06.059081 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:40:06.059486 (+ 405us) client_negotiation.cc:769] Sending connection context
0925 22:40:06.059696 (+ 210us) client_negotiation.cc:240] Negotiation successful
0925 22:40:06.060104 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":106,"threads_started":1}
I20260925 22:40:06.063062 25461 heartbeater.cc:347] Connected to a master server at 127.6.246.254:33155
I20260925 22:40:06.063345 25461 heartbeater.cc:464] Registering TS with master...
I20260925 22:40:06.063848 25461 heartbeater.cc:511] Master 127.6.246.254:33155 requested a full tablet report, sending...
I20260925 22:40:06.065593 25268 ts_manager.cc:194] Registered new tserver with Master: d44e1943717546bc866473b91e63a64f (127.6.246.193:35911)
I20260925 22:40:06.067307 25268 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.246.193@KRBTEST.COM'} at 127.6.246.193:43691
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.194.keytab.
Entry for principal kudu/127.6.246.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.194.keytab.
I20260925 22:40:06.079149 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.194:0
--local_ip_for_outbound_sockets=127.6.246.194
--webserver_interface=127.6.246.194
--webserver_port=0
--tserver_master_addrs=127.6.246.254:33155
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.194.keytab
--principal=kudu/127.6.246.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:40715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:40:06.188220 25466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:40:06.188521 25466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:40:06.188589 25466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:40:06.192433 25466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:40:06.192556 25466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:40:06.192618 25466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.194
I20260925 22:40:06.197801 25466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:40715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.194.keytab
--principal=kudu/127.6.246.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.246.194
--webserver_port=0
--tserver_master_addrs=127.6.246.254:33155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.194
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:40:06.198881 25466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:40:06.199960 25466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:40:06.209851 25471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:06.209846 25474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:06.209864 25472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:40:06.210283 25466 server_base.cc:1034] running on GCE node
I20260925 22:40:06.210727 25466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:40:06.211359 25466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:40:06.212574 25466 hybrid_clock.cc:648] HybridClock initialized: now 1790376006212543 us; error 44 us; skew 500 ppm
Sep 25 22:40:06 dist-test-slave-wjdr krb5kdc[25225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376006, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:40:06.215379 25466 init.cc:377] Logged in from keytab as kudu/127.6.246.194@KRBTEST.COM (short username kudu)
I20260925 22:40:06.216270 25466 webserver.cc:460] Webserver started at http://127.6.246.194:45859/ using document root <none> and password file <none>
I20260925 22:40:06.216845 25466 fs_manager.cc:359] Metadata directory not provided
I20260925 22:40:06.216905 25466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:40:06.217063 25466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:40:06.218979 25466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/minicluster-data/ts-1/data/instance:
uuid: "902217bf42594df99d3b3e5a7a34ab84"
format_stamp: "Formatted at 2026-09-25 22:40:06 on dist-test-slave-wjdr"
server_key: "ef22080d87d4e5ff90f0620f082b9643"
server_key_iv: "a2628ae4bac013cd5b759eedb9c7addb"
server_key_version: "encryptionkey@0"
I20260925 22:40:06.219473 25466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/minicluster-data/ts-1/wal/instance:
uuid: "902217bf42594df99d3b3e5a7a34ab84"
format_stamp: "Formatted at 2026-09-25 22:40:06 on dist-test-slave-wjdr"
server_key: "ef22080d87d4e5ff90f0620f082b9643"
server_key_iv: "a2628ae4bac013cd5b759eedb9c7addb"
server_key_version: "encryptionkey@0"
I20260925 22:40:06.223008 25466 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 22:40:06.225360 25481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:06.226536 25466 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:40:06.226657 25466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/minicluster-data/ts-1/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/minicluster-data/ts-1/wal
uuid: "902217bf42594df99d3b3e5a7a34ab84"
format_stamp: "Formatted at 2026-09-25 22:40:06 on dist-test-slave-wjdr"
server_key: "ef22080d87d4e5ff90f0620f082b9643"
server_key_iv: "a2628ae4bac013cd5b759eedb9c7addb"
server_key_version: "encryptionkey@0"
I20260925 22:40:06.226761 25466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:40:06.246539 25466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:40:06.249615 25466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:40:06.249835 25466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:40:06.250488 25466 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:40:06.251325 25466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:40:06.251372 25466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:06.251412 25466 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:40:06.251428 25466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:06.260747 25466 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.194:45211
I20260925 22:40:06.260772 25594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.194:45211 every 8 connection(s)
I20260925 22:40:06.261664 25466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
I20260925 22:40:06.265615 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 25466
I20260925 22:40:06.265800 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/minicluster-data/ts-1/wal/instance
I20260925 22:40:06.266129 7131 external_mini_cluster.cc:1428] Setting key c5082227adfecfd5bada48252201bc69
Sep 25 22:40:06 dist-test-slave-wjdr krb5kdc[25225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376006, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.194@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.246.195@KRBTEST.COM; defaulting to no policy
I20260925 22:40:06.276259 25309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:06.263547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.194:40219 (local address 127.6.246.254:33155)
0925 22:40:06.263730 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:06.263735 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:06.264627 (+ 892us) server_negotiation.cc:407] Connection header received
0925 22:40:06.265485 (+ 858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:06.265492 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:06.265555 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:06.265660 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:06.267110 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:06.267656 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:06.268896 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:06.269124 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:06.269242 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:06.272170 (+ 2928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:06.272176 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:06.272180 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:06.272215 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:06.274062 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:06.274584 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:06.274589 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:06.274591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:06.274688 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:06.275032 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:06.275037 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:06.275041 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:06.275297 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:06.275426 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:06.275903 (+ 477us) server_negotiation.cc:299] Negotiation successful
0925 22:40:06.276044 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 22:40:06.277097 25597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:06.263973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:33155 (local address 127.6.246.194:40219)
0925 22:40:06.264425 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:06.264488 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:06.265271 (+ 783us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:06.265897 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:06.265906 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:06.266337 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:40:06.266953 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:06.266969 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:06.267796 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:06.267800 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:06.268764 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:06.268773 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:06.269278 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:06.269283 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:06.269463 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:06.270088 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:40:06.270104 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:40:06.271953 (+ 1849us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:40:06.274195 (+ 2242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:06.274200 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:06.274212 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:06.274469 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:06.274790 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:06.274798 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:06.274799 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:06.274926 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:06.275421 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:40:06.275426 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:40:06.275651 (+ 225us) client_negotiation.cc:769] Sending connection context
0925 22:40:06.275845 (+ 194us) client_negotiation.cc:240] Negotiation successful
0925 22:40:06.276148 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":114,"threads_started":1}
I20260925 22:40:06.278455 25595 heartbeater.cc:347] Connected to a master server at 127.6.246.254:33155
I20260925 22:40:06.278697 25595 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.195@KRBTEST.COM" created.
I20260925 22:40:06.279249 25595 heartbeater.cc:511] Master 127.6.246.254:33155 requested a full tablet report, sending...
I20260925 22:40:06.280337 25268 ts_manager.cc:194] Registered new tserver with Master: 902217bf42594df99d3b3e5a7a34ab84 (127.6.246.194:45211)
I20260925 22:40:06.281235 25268 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.246.194@KRBTEST.COM'} at 127.6.246.194:40219
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.195.keytab.
Entry for principal kudu/127.6.246.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.195.keytab.
I20260925 22:40:06.296028 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.195:0
--local_ip_for_outbound_sockets=127.6.246.195
--webserver_interface=127.6.246.195
--webserver_port=0
--tserver_master_addrs=127.6.246.254:33155
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.195.keytab
--principal=kudu/127.6.246.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:40715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:40:06.416128 25600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:40:06.416594 25600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:40:06.416666 25600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:40:06.421012 25600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:40:06.421094 25600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:40:06.421162 25600 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.195
I20260925 22:40:06.426965 25600 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:40715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.195.keytab
--principal=kudu/127.6.246.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.246.195
--webserver_port=0
--tserver_master_addrs=127.6.246.254:33155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25600
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.195
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:40:06.428375 25600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:40:06.429760 25600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:40:06.442010 25605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:06.442023 25608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:06.442010 25606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:40:06.442461 25600 server_base.cc:1034] running on GCE node
I20260925 22:40:06.442961 25600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:40:06.443785 25600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:40:06.445029 25600 hybrid_clock.cc:648] HybridClock initialized: now 1790376006445004 us; error 69 us; skew 500 ppm
Sep 25 22:40:06 dist-test-slave-wjdr krb5kdc[25225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376006, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:40:06.448251 25600 init.cc:377] Logged in from keytab as kudu/127.6.246.195@KRBTEST.COM (short username kudu)
I20260925 22:40:06.449404 25600 webserver.cc:460] Webserver started at http://127.6.246.195:39955/ using document root <none> and password file <none>
I20260925 22:40:06.450114 25600 fs_manager.cc:359] Metadata directory not provided
I20260925 22:40:06.450214 25600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:40:06.450477 25600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:40:06.453133 25600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/minicluster-data/ts-2/data/instance:
uuid: "1fb59d2d608448ee8e548f9c3e522482"
format_stamp: "Formatted at 2026-09-25 22:40:06 on dist-test-slave-wjdr"
server_key: "05bb15d8a2a5dcbef2aa8ddff3699058"
server_key_iv: "5b89f5215c2ce38b28988c30bdb8e841"
server_key_version: "encryptionkey@0"
I20260925 22:40:06.453799 25600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/minicluster-data/ts-2/wal/instance:
uuid: "1fb59d2d608448ee8e548f9c3e522482"
format_stamp: "Formatted at 2026-09-25 22:40:06 on dist-test-slave-wjdr"
server_key: "05bb15d8a2a5dcbef2aa8ddff3699058"
server_key_iv: "5b89f5215c2ce38b28988c30bdb8e841"
server_key_version: "encryptionkey@0"
I20260925 22:40:06.458199 25600 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:40:06.461136 25615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:06.462602 25600 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:40:06.462762 25600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/minicluster-data/ts-2/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/minicluster-data/ts-2/wal
uuid: "1fb59d2d608448ee8e548f9c3e522482"
format_stamp: "Formatted at 2026-09-25 22:40:06 on dist-test-slave-wjdr"
server_key: "05bb15d8a2a5dcbef2aa8ddff3699058"
server_key_iv: "5b89f5215c2ce38b28988c30bdb8e841"
server_key_version: "encryptionkey@0"
I20260925 22:40:06.462896 25600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:40:06.485149 25600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:40:06.488907 25600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:40:06.489143 25600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:40:06.489876 25600 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:40:06.490866 25600 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:40:06.490945 25600 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:06.491022 25600 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:40:06.491055 25600 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:06.501310 25600 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.195:38759
I20260925 22:40:06.501389 25728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.195:38759 every 8 connection(s)
I20260925 22:40:06.502434 25600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
I20260925 22:40:06.503685 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 25600
I20260925 22:40:06.503863 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/minicluster-data/ts-2/wal/instance
I20260925 22:40:06.504271 7131 external_mini_cluster.cc:1428] Setting key 2f913ff2888ff694d880a7f5d943ba72
Sep 25 22:40:06 dist-test-slave-wjdr krb5kdc[25225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376006, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.195@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
I20260925 22:40:06.517786 25309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:06.504901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.195:43685 (local address 127.6.246.254:33155)
0925 22:40:06.505084 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:06.505088 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:06.506332 (+ 1244us) server_negotiation.cc:407] Connection header received
0925 22:40:06.507455 (+ 1123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:06.507458 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:06.507509 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:06.507583 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:06.508942 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:06.509527 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:06.510788 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:06.511020 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:06.511149 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:06.514064 (+ 2915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:06.514067 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:06.514070 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:06.514095 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:06.515461 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:06.516002 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:06.516007 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:06.516009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:06.516105 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:06.516480 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:06.516487 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:06.516491 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:06.516698 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:06.516817 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:06.517455 (+ 638us) server_negotiation.cc:299] Negotiation successful
0925 22:40:06.517598 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 22:40:06.518987 25731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:06.505444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:33155 (local address 127.6.246.195:43685)
0925 22:40:06.506140 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:06.506208 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:06.507249 (+ 1041us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:06.507731 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:06.507738 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:06.508156 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:40:06.508805 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:06.508821 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:06.509683 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:06.509686 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:06.510649 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:06.510660 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:06.511193 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:06.511198 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:06.511386 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:06.512055 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:40:06.512074 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:40:06.513864 (+ 1790us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:40:06.515593 (+ 1729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:06.515599 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:06.515616 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:06.515888 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:06.516207 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:06.516211 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:06.516213 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:06.516361 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:06.516811 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:40:06.516817 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:40:06.517152 (+ 335us) client_negotiation.cc:769] Sending connection context
0925 22:40:06.517404 (+ 252us) client_negotiation.cc:240] Negotiation successful
0925 22:40:06.517792 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":192,"threads_started":1}
I20260925 22:40:06.520215 25729 heartbeater.cc:347] Connected to a master server at 127.6.246.254:33155
I20260925 22:40:06.520566 25729 heartbeater.cc:464] Registering TS with master...
I20260925 22:40:06.521216 25729 heartbeater.cc:511] Master 127.6.246.254:33155 requested a full tablet report, sending...
I20260925 22:40:06.522368 25268 ts_manager.cc:194] Registered new tserver with Master: 1fb59d2d608448ee8e548f9c3e522482 (127.6.246.195:38759)
I20260925 22:40:06.523020 25268 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.246.195@KRBTEST.COM'} at 127.6.246.195:43685
I20260925 22:40:06.531159 7131 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:40:06.541462 25309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:06.533653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57482 (local address 127.6.246.254:33155)
0925 22:40:06.533825 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:06.533829 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:06.534038 (+ 209us) server_negotiation.cc:407] Connection header received
0925 22:40:06.534113 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:06.534115 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:06.534165 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:06.534240 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:06.535009 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:06.535605 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:06.536472 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:06.536690 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:06.536769 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:06.538458 (+ 1689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:06.538461 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:06.538463 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:06.538488 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:06.539765 (+ 1277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:06.540225 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:06.540227 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:06.540229 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:06.540300 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:06.540667 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:06.540669 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:06.540670 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:06.540886 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:06.541016 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:06.541216 (+ 200us) server_negotiation.cc:299] Negotiation successful
0925 22:40:06.541296 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 22:40:06.545034 25268 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57482:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:40:06.548856 25268 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:40:06.565990 25744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:06.559261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.194:45211 (local address 127.0.0.1:38460)
0925 22:40:06.560295 (+ 1034us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:06.560358 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:06.560534 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:06.561753 (+ 1219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:06.561756 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:06.561774 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:40:06.561882 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:06.561889 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:06.563423 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:06.563427 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:06.564596 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:06.564607 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:06.565532 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:06.565536 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:06.565631 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:06.565653 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 22:40:06.565729 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 22:40:06.565809 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":842,"thread_start_us":118,"threads_started":1}
I20260925 22:40:06.566033 25745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:06.559938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38460 (local address 127.6.246.194:45211)
0925 22:40:06.560484 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:06.560511 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:06.560540 (+ 29us) server_negotiation.cc:407] Connection header received
0925 22:40:06.561397 (+ 857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:06.561403 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:06.561589 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:06.562021 (+ 432us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:40:06.562182 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:06.563269 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:06.564724 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:06.565399 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:06.565510 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:06.565645 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:06.565758 (+ 113us) server_negotiation.cc:299] Negotiation successful
0925 22:40:06.565865 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":182,"threads_started":1}
I20260925 22:40:06.566350 25743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:06.559353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56936 (local address 127.6.246.193:35911)
0925 22:40:06.560021 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:06.560039 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:06.560859 (+ 820us) server_negotiation.cc:407] Connection header received
0925 22:40:06.560947 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:06.560951 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:06.561133 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:06.561285 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:40:06.561483 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:06.562755 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:06.564623 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:06.565430 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:06.565562 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:06.565724 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:06.566081 (+ 357us) server_negotiation.cc:299] Negotiation successful
0925 22:40:06.566184 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":84,"threads_started":1}
I20260925 22:40:06.566520 25742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:06.559275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52154 (local address 127.6.246.195:38759)
0925 22:40:06.559989 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:06.560002 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:06.560604 (+ 602us) server_negotiation.cc:407] Connection header received
0925 22:40:06.560712 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:06.560716 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:06.560968 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:06.561142 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:40:06.561536 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:06.562432 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:06.564175 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:06.565040 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:06.565169 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:06.565341 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:06.565911 (+ 570us) server_negotiation.cc:299] Negotiation successful
0925 22:40:06.566036 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":122,"threads_started":1}
I20260925 22:40:06.566900 25740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:06.559061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.193:35911 (local address 127.0.0.1:56936)
0925 22:40:06.560755 (+ 1694us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:06.560766 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:06.560841 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:06.561288 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:06.561291 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:06.561303 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:40:06.561388 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:06.561393 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:06.562896 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:06.562899 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:06.564401 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:06.564411 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:06.565783 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:06.565786 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:06.565870 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:06.565948 (+ 78us) client_negotiation.cc:769] Sending connection context
0925 22:40:06.566662 (+ 714us) client_negotiation.cc:240] Negotiation successful
0925 22:40:06.566753 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1641,"mutex_wait_us":547,"thread_start_us":111,"threads_started":1}
I20260925 22:40:06.565840 25741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:06.559061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.195:38759 (local address 127.0.0.1:52154)
0925 22:40:06.560295 (+ 1234us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:06.560358 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:06.560534 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:06.561147 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:06.561153 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:06.561173 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:40:06.561386 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:06.561393 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:06.562576 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:06.562582 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:06.564034 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:06.564043 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:06.565195 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:06.565198 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:06.565311 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:06.565357 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 22:40:06.565488 (+ 131us) client_negotiation.cc:240] Negotiation successful
0925 22:40:06.565631 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1025,"mutex_wait_us":168,"spinlock_wait_cycles":25856,"thread_start_us":119,"threads_started":1}
I20260925 22:40:06.570173 25395 tablet_service.cc:1467] Processing CreateTablet for tablet 4518087061c34feeb58acf1c7b9925c3 (DEFAULT_TABLE table=test-table [id=02144f290c924357ab0dbe66462cef4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:40:06.570183 25663 tablet_service.cc:1467] Processing CreateTablet for tablet 4518087061c34feeb58acf1c7b9925c3 (DEFAULT_TABLE table=test-table [id=02144f290c924357ab0dbe66462cef4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:40:06.570184 25529 tablet_service.cc:1467] Processing CreateTablet for tablet 4518087061c34feeb58acf1c7b9925c3 (DEFAULT_TABLE table=test-table [id=02144f290c924357ab0dbe66462cef4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:40:06.571470 25395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4518087061c34feeb58acf1c7b9925c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:40:06.571471 25529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4518087061c34feeb58acf1c7b9925c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:40:06.571815 25663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4518087061c34feeb58acf1c7b9925c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:40:06.579063 25746 tablet_bootstrap.cc:492] T 4518087061c34feeb58acf1c7b9925c3 P d44e1943717546bc866473b91e63a64f: Bootstrap starting.
I20260925 22:40:06.580814 25747 tablet_bootstrap.cc:492] T 4518087061c34feeb58acf1c7b9925c3 P 1fb59d2d608448ee8e548f9c3e522482: Bootstrap starting.
I20260925 22:40:06.581241 25748 tablet_bootstrap.cc:492] T 4518087061c34feeb58acf1c7b9925c3 P 902217bf42594df99d3b3e5a7a34ab84: Bootstrap starting.
I20260925 22:40:06.582089 25746 tablet_bootstrap.cc:654] T 4518087061c34feeb58acf1c7b9925c3 P d44e1943717546bc866473b91e63a64f: Neither blocks nor log segments found. Creating new log.
I20260925 22:40:06.583114 25746 log.cc:824] T 4518087061c34feeb58acf1c7b9925c3 P d44e1943717546bc866473b91e63a64f: Log is configured to *not* fsync() on all Append() calls
I20260925 22:40:06.583539 25748 tablet_bootstrap.cc:654] T 4518087061c34feeb58acf1c7b9925c3 P 902217bf42594df99d3b3e5a7a34ab84: Neither blocks nor log segments found. Creating new log.
I20260925 22:40:06.584818 25748 log.cc:824] T 4518087061c34feeb58acf1c7b9925c3 P 902217bf42594df99d3b3e5a7a34ab84: Log is configured to *not* fsync() on all Append() calls
I20260925 22:40:06.585793 25747 tablet_bootstrap.cc:654] T 4518087061c34feeb58acf1c7b9925c3 P 1fb59d2d608448ee8e548f9c3e522482: Neither blocks nor log segments found. Creating new log.
I20260925 22:40:06.585819 25746 tablet_bootstrap.cc:492] T 4518087061c34feeb58acf1c7b9925c3 P d44e1943717546bc866473b91e63a64f: No bootstrap required, opened a new log
I20260925 22:40:06.586067 25746 ts_tablet_manager.cc:1397] T 4518087061c34feeb58acf1c7b9925c3 P d44e1943717546bc866473b91e63a64f: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 22:40:06.586820 25748 tablet_bootstrap.cc:492] T 4518087061c34feeb58acf1c7b9925c3 P 902217bf42594df99d3b3e5a7a34ab84: No bootstrap required, opened a new log
I20260925 22:40:06.587059 25748 ts_tablet_manager.cc:1397] T 4518087061c34feeb58acf1c7b9925c3 P 902217bf42594df99d3b3e5a7a34ab84: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260925 22:40:06.587155 25747 log.cc:824] T 4518087061c34feeb58acf1c7b9925c3 P 1fb59d2d608448ee8e548f9c3e522482: Log is configured to *not* fsync() on all Append() calls
I20260925 22:40:06.589457 25747 tablet_bootstrap.cc:492] T 4518087061c34feeb58acf1c7b9925c3 P 1fb59d2d608448ee8e548f9c3e522482: No bootstrap required, opened a new log
I20260925 22:40:06.589687 25747 ts_tablet_manager.cc:1397] T 4518087061c34feeb58acf1c7b9925c3 P 1fb59d2d608448ee8e548f9c3e522482: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260925 22:40:06.590718 25748 raft_consensus.cc:348] T 4518087061c34feeb58acf1c7b9925c3 P 902217bf42594df99d3b3e5a7a34ab84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fb59d2d608448ee8e548f9c3e522482" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 38759 } } peers { permanent_uuid: "d44e1943717546bc866473b91e63a64f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 35911 } } peers { permanent_uuid: "902217bf42594df99d3b3e5a7a34ab84" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 45211 } }
I20260925 22:40:06.591014 25748 raft_consensus.cc:374] T 4518087061c34feeb58acf1c7b9925c3 P 902217bf42594df99d3b3e5a7a34ab84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:40:06.591090 25748 raft_consensus.cc:729] T 4518087061c34feeb58acf1c7b9925c3 P 902217bf42594df99d3b3e5a7a34ab84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 902217bf42594df99d3b3e5a7a34ab84, State: Initialized, Role: FOLLOWER
I20260925 22:40:06.591614 25748 consensus_queue.cc:260] T 4518087061c34feeb58acf1c7b9925c3 P 902217bf42594df99d3b3e5a7a34ab84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fb59d2d608448ee8e548f9c3e522482" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 38759 } } peers { permanent_uuid: "d44e1943717546bc866473b91e63a64f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 35911 } } peers { permanent_uuid: "902217bf42594df99d3b3e5a7a34ab84" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 45211 } }
I20260925 22:40:06.591984 25746 raft_consensus.cc:348] T 4518087061c34feeb58acf1c7b9925c3 P d44e1943717546bc866473b91e63a64f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fb59d2d608448ee8e548f9c3e522482" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 38759 } } peers { permanent_uuid: "d44e1943717546bc866473b91e63a64f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 35911 } } peers { permanent_uuid: "902217bf42594df99d3b3e5a7a34ab84" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 45211 } }
I20260925 22:40:06.592329 25746 raft_consensus.cc:374] T 4518087061c34feeb58acf1c7b9925c3 P d44e1943717546bc866473b91e63a64f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:40:06.592387 25595 heartbeater.cc:503] Master 127.6.246.254:33155 was elected leader, sending a full tablet report...
I20260925 22:40:06.592435 25746 raft_consensus.cc:729] T 4518087061c34feeb58acf1c7b9925c3 P d44e1943717546bc866473b91e63a64f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d44e1943717546bc866473b91e63a64f, State: Initialized, Role: FOLLOWER
I20260925 22:40:06.592903 25748 ts_tablet_manager.cc:1428] T 4518087061c34feeb58acf1c7b9925c3 P 902217bf42594df99d3b3e5a7a34ab84: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 22:40:06.593016 25746 consensus_queue.cc:260] T 4518087061c34feeb58acf1c7b9925c3 P d44e1943717546bc866473b91e63a64f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fb59d2d608448ee8e548f9c3e522482" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 38759 } } peers { permanent_uuid: "d44e1943717546bc866473b91e63a64f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 35911 } } peers { permanent_uuid: "902217bf42594df99d3b3e5a7a34ab84" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 45211 } }
I20260925 22:40:06.593863 25461 heartbeater.cc:503] Master 127.6.246.254:33155 was elected leader, sending a full tablet report...
I20260925 22:40:06.593842 25747 raft_consensus.cc:348] T 4518087061c34feeb58acf1c7b9925c3 P 1fb59d2d608448ee8e548f9c3e522482 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fb59d2d608448ee8e548f9c3e522482" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 38759 } } peers { permanent_uuid: "d44e1943717546bc866473b91e63a64f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 35911 } } peers { permanent_uuid: "902217bf42594df99d3b3e5a7a34ab84" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 45211 } }
I20260925 22:40:06.594177 25747 raft_consensus.cc:374] T 4518087061c34feeb58acf1c7b9925c3 P 1fb59d2d608448ee8e548f9c3e522482 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:40:06.594247 25746 ts_tablet_manager.cc:1428] T 4518087061c34feeb58acf1c7b9925c3 P d44e1943717546bc866473b91e63a64f: Time spent starting tablet: real 0.008s user 0.004s sys 0.005s
I20260925 22:40:06.594276 25747 raft_consensus.cc:729] T 4518087061c34feeb58acf1c7b9925c3 P 1fb59d2d608448ee8e548f9c3e522482 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fb59d2d608448ee8e548f9c3e522482, State: Initialized, Role: FOLLOWER
I20260925 22:40:06.594805 25747 consensus_queue.cc:260] T 4518087061c34feeb58acf1c7b9925c3 P 1fb59d2d608448ee8e548f9c3e522482 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fb59d2d608448ee8e548f9c3e522482" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 38759 } } peers { permanent_uuid: "d44e1943717546bc866473b91e63a64f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 35911 } } peers { permanent_uuid: "902217bf42594df99d3b3e5a7a34ab84" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 45211 } }
I20260925 22:40:06.595602 25729 heartbeater.cc:503] Master 127.6.246.254:33155 was elected leader, sending a full tablet report...
I20260925 22:40:06.595865 25747 ts_tablet_manager.cc:1428] T 4518087061c34feeb58acf1c7b9925c3 P 1fb59d2d608448ee8e548f9c3e522482: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 22:40:06.731050 25753 raft_consensus.cc:482] T 4518087061c34feeb58acf1c7b9925c3 P d44e1943717546bc866473b91e63a64f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:40:06.731413 25753 raft_consensus.cc:504] T 4518087061c34feeb58acf1c7b9925c3 P d44e1943717546bc866473b91e63a64f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fb59d2d608448ee8e548f9c3e522482" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 38759 } } peers { permanent_uuid: "d44e1943717546bc866473b91e63a64f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 35911 } } peers { permanent_uuid: "902217bf42594df99d3b3e5a7a34ab84" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 45211 } }
I20260925 22:40:06.733160 25753 leader_election.cc:288] T 4518087061c34feeb58acf1c7b9925c3 P d44e1943717546bc866473b91e63a64f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1fb59d2d608448ee8e548f9c3e522482 (127.6.246.195:38759), 902217bf42594df99d3b3e5a7a34ab84 (127.6.246.194:45211)
I20260925 22:40:06.737744 25745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:06.733752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.193:53881 (local address 127.6.246.194:45211)
0925 22:40:06.733906 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:06.733911 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:06.734100 (+ 189us) server_negotiation.cc:407] Connection header received
0925 22:40:06.734291 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:06.734294 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:06.734345 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:06.734441 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:40:06.734809 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:06.735392 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:06.736686 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:06.737115 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:06.737250 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:06.737316 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:06.737501 (+ 185us) server_negotiation.cc:299] Negotiation successful
0925 22:40:06.737587 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 22:40:06.737746 25742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:06.733600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.193:50217 (local address 127.6.246.195:38759)
0925 22:40:06.733819 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:06.733825 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:06.734074 (+ 249us) server_negotiation.cc:407] Connection header received
0925 22:40:06.734289 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:06.734292 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:06.734340 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:06.734434 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:40:06.734808 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:06.735402 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:06.736696 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:06.737130 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:06.737269 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:06.737329 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:06.737505 (+ 176us) server_negotiation.cc:299] Negotiation successful
0925 22:40:06.737591 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 22:40:06.737771 25755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:06.733454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.195:38759 (local address 127.6.246.193:50217)
0925 22:40:06.733965 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:06.733979 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:06.734161 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:06.734446 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:06.734448 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:06.734487 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:40:06.734692 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:06.734699 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:06.735514 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:06.735520 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:06.736575 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:06.736584 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:06.737256 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:06.737259 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:06.737368 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:06.737389 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 22:40:06.737489 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 22:40:06.737611 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"mutex_wait_us":222,"spinlock_wait_cycles":44544,"thread_start_us":163,"threads_started":1}
I20260925 22:40:06.737775 25756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:06.733488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.194:45211 (local address 127.6.246.193:53881)
0925 22:40:06.733988 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:06.734000 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:06.734169 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:06.734447 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:06.734450 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:06.734489 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:40:06.734695 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:06.734701 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:06.735513 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:06.735519 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:06.736562 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:06.736578 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:06.737242 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:06.737245 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:06.737343 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:06.737377 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 22:40:06.737478 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 22:40:06.737587 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"mutex_wait_us":82,"spinlock_wait_cycles":23936,"thread_start_us":146,"threads_started":1}
I20260925 22:40:06.738519 25683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4518087061c34feeb58acf1c7b9925c3" candidate_uuid: "d44e1943717546bc866473b91e63a64f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fb59d2d608448ee8e548f9c3e522482" is_pre_election: true
I20260925 22:40:06.738667 25549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4518087061c34feeb58acf1c7b9925c3" candidate_uuid: "d44e1943717546bc866473b91e63a64f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "902217bf42594df99d3b3e5a7a34ab84" is_pre_election: true
I20260925 22:40:06.738927 25683 raft_consensus.cc:2454] T 4518087061c34feeb58acf1c7b9925c3 P 1fb59d2d608448ee8e548f9c3e522482 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d44e1943717546bc866473b91e63a64f in term 0.
I20260925 22:40:06.739022 25549 raft_consensus.cc:2454] T 4518087061c34feeb58acf1c7b9925c3 P 902217bf42594df99d3b3e5a7a34ab84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d44e1943717546bc866473b91e63a64f in term 0.
I20260925 22:40:06.739544 25351 leader_election.cc:302] T 4518087061c34feeb58acf1c7b9925c3 P d44e1943717546bc866473b91e63a64f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1fb59d2d608448ee8e548f9c3e522482, 902217bf42594df99d3b3e5a7a34ab84, d44e1943717546bc866473b91e63a64f; no voters:
I20260925 22:40:06.739902 25753 raft_consensus.cc:2781] T 4518087061c34feeb58acf1c7b9925c3 P d44e1943717546bc866473b91e63a64f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:40:06.739998 25753 raft_consensus.cc:482] T 4518087061c34feeb58acf1c7b9925c3 P d44e1943717546bc866473b91e63a64f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:40:06.740057 25753 raft_consensus.cc:3037] T 4518087061c34feeb58acf1c7b9925c3 P d44e1943717546bc866473b91e63a64f [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:40:06.741413 25753 raft_consensus.cc:504] T 4518087061c34feeb58acf1c7b9925c3 P d44e1943717546bc866473b91e63a64f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fb59d2d608448ee8e548f9c3e522482" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 38759 } } peers { permanent_uuid: "d44e1943717546bc866473b91e63a64f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 35911 } } peers { permanent_uuid: "902217bf42594df99d3b3e5a7a34ab84" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 45211 } }
I20260925 22:40:06.741835 25753 leader_election.cc:288] T 4518087061c34feeb58acf1c7b9925c3 P d44e1943717546bc866473b91e63a64f [CANDIDATE]: Term 1 election: Requested vote from peers 1fb59d2d608448ee8e548f9c3e522482 (127.6.246.195:38759), 902217bf42594df99d3b3e5a7a34ab84 (127.6.246.194:45211)
I20260925 22:40:06.742316 25549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4518087061c34feeb58acf1c7b9925c3" candidate_uuid: "d44e1943717546bc866473b91e63a64f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "902217bf42594df99d3b3e5a7a34ab84"
I20260925 22:40:06.742316 25683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4518087061c34feeb58acf1c7b9925c3" candidate_uuid: "d44e1943717546bc866473b91e63a64f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fb59d2d608448ee8e548f9c3e522482"
I20260925 22:40:06.742479 25683 raft_consensus.cc:3037] T 4518087061c34feeb58acf1c7b9925c3 P 1fb59d2d608448ee8e548f9c3e522482 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:40:06.742479 25549 raft_consensus.cc:3037] T 4518087061c34feeb58acf1c7b9925c3 P 902217bf42594df99d3b3e5a7a34ab84 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:40:06.744095 25549 raft_consensus.cc:2454] T 4518087061c34feeb58acf1c7b9925c3 P 902217bf42594df99d3b3e5a7a34ab84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d44e1943717546bc866473b91e63a64f in term 1.
I20260925 22:40:06.744095 25683 raft_consensus.cc:2454] T 4518087061c34feeb58acf1c7b9925c3 P 1fb59d2d608448ee8e548f9c3e522482 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d44e1943717546bc866473b91e63a64f in term 1.
I20260925 22:40:06.744553 25349 leader_election.cc:302] T 4518087061c34feeb58acf1c7b9925c3 P d44e1943717546bc866473b91e63a64f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 902217bf42594df99d3b3e5a7a34ab84, d44e1943717546bc866473b91e63a64f; no voters:
I20260925 22:40:06.744830 25753 raft_consensus.cc:2781] T 4518087061c34feeb58acf1c7b9925c3 P d44e1943717546bc866473b91e63a64f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:40:06.745178 25753 raft_consensus.cc:686] T 4518087061c34feeb58acf1c7b9925c3 P d44e1943717546bc866473b91e63a64f [term 1 LEADER]: Becoming Leader. State: Replica: d44e1943717546bc866473b91e63a64f, State: Running, Role: LEADER
I20260925 22:40:06.745675 25753 consensus_queue.cc:237] T 4518087061c34feeb58acf1c7b9925c3 P d44e1943717546bc866473b91e63a64f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fb59d2d608448ee8e548f9c3e522482" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 38759 } } peers { permanent_uuid: "d44e1943717546bc866473b91e63a64f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 35911 } } peers { permanent_uuid: "902217bf42594df99d3b3e5a7a34ab84" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 45211 } }
I20260925 22:40:06.749543 25269 catalog_manager.cc:5495] T 4518087061c34feeb58acf1c7b9925c3 P d44e1943717546bc866473b91e63a64f reported cstate change: term changed from 0 to 1, leader changed from <none> to d44e1943717546bc866473b91e63a64f (127.6.246.193). New cstate: current_term: 1 leader_uuid: "d44e1943717546bc866473b91e63a64f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fb59d2d608448ee8e548f9c3e522482" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 38759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d44e1943717546bc866473b91e63a64f" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 35911 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "902217bf42594df99d3b3e5a7a34ab84" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 45211 } health_report { overall_health: UNKNOWN } } }
W20260925 22:40:06.753544 25730 tablet.cc:2369] T 4518087061c34feeb58acf1c7b9925c3 P 1fb59d2d608448ee8e548f9c3e522482: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 22:40:06.763020 25596 tablet.cc:2369] T 4518087061c34feeb58acf1c7b9925c3 P 902217bf42594df99d3b3e5a7a34ab84: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 22:40:06.787834 25462 tablet.cc:2369] T 4518087061c34feeb58acf1c7b9925c3 P d44e1943717546bc866473b91e63a64f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:40:06.808418 25743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:06.804294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56942 (local address 127.6.246.193:35911)
0925 22:40:06.804445 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:06.804495 (+ 50us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:06.804513 (+ 18us) server_negotiation.cc:407] Connection header received
0925 22:40:06.804620 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:06.804623 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:06.804689 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:06.804787 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:40:06.805075 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:06.805629 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:06.806592 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:06.806856 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:06.806946 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:06.807325 (+ 379us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:40:06.807876 (+ 551us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:40:06.807997 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:06.808134 (+ 137us) server_negotiation.cc:299] Negotiation successful
0925 22:40:06.808208 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 22:40:06.816545 25683 raft_consensus.cc:1261] T 4518087061c34feeb58acf1c7b9925c3 P 1fb59d2d608448ee8e548f9c3e522482 [term 1 FOLLOWER]: Refusing update from remote peer d44e1943717546bc866473b91e63a64f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:40:06.816545 25549 raft_consensus.cc:1261] T 4518087061c34feeb58acf1c7b9925c3 P 902217bf42594df99d3b3e5a7a34ab84 [term 1 FOLLOWER]: Refusing update from remote peer d44e1943717546bc866473b91e63a64f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:40:06.817510 25753 consensus_queue.cc:1035] T 4518087061c34feeb58acf1c7b9925c3 P d44e1943717546bc866473b91e63a64f [LEADER]: Connected to new peer: Peer: permanent_uuid: "902217bf42594df99d3b3e5a7a34ab84" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 45211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:40:06.817833 25757 consensus_queue.cc:1035] T 4518087061c34feeb58acf1c7b9925c3 P d44e1943717546bc866473b91e63a64f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fb59d2d608448ee8e548f9c3e522482" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 38759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:40:06.826988 25760 mvcc.cc:205] Tried to move back new op lower bound from 7333380123909357568 to 7333380123633487872. Current Snapshot: MvccSnapshot[applied={T|T < 7333380123909357568}]
I20260925 22:40:06.828035 25762 mvcc.cc:205] Tried to move back new op lower bound from 7333380123909357568 to 7333380123633487872. Current Snapshot: MvccSnapshot[applied={T|T < 7333380123909357568}]
I20260925 22:40:06.838820 25269 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57482:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:40:06.839089 25269 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57482:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:40:06.842515 25269 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1d1ceac58a6043c1a093a43d49a95994: Sending DeleteTablet for 3 replicas of tablet 4518087061c34feeb58acf1c7b9925c3
I20260925 22:40:06.843638 25395 tablet_service.cc:1514] Processing DeleteTablet for tablet 4518087061c34feeb58acf1c7b9925c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:40:06 UTC) from {username='kudu', principal='kudu/127.6.246.254@KRBTEST.COM'} at 127.0.0.1:56936
I20260925 22:40:06.843798 25663 tablet_service.cc:1514] Processing DeleteTablet for tablet 4518087061c34feeb58acf1c7b9925c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:40:06 UTC) from {username='kudu', principal='kudu/127.6.246.254@KRBTEST.COM'} at 127.0.0.1:52154
I20260925 22:40:06.844063 25529 tablet_service.cc:1514] Processing DeleteTablet for tablet 4518087061c34feeb58acf1c7b9925c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:40:06 UTC) from {username='kudu', principal='kudu/127.6.246.254@KRBTEST.COM'} at 127.0.0.1:38460
I20260925 22:40:06.844597 25770 tablet_replica.cc:330] T 4518087061c34feeb58acf1c7b9925c3 P d44e1943717546bc866473b91e63a64f: stopping tablet replica
I20260925 22:40:06.844798 25771 tablet_replica.cc:330] T 4518087061c34feeb58acf1c7b9925c3 P 1fb59d2d608448ee8e548f9c3e522482: stopping tablet replica
I20260925 22:40:06.845099 25770 raft_consensus.cc:2229] T 4518087061c34feeb58acf1c7b9925c3 P d44e1943717546bc866473b91e63a64f [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:40:06.845296 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 25332
I20260925 22:40:06.845283 25771 raft_consensus.cc:2229] T 4518087061c34feeb58acf1c7b9925c3 P 1fb59d2d608448ee8e548f9c3e522482 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:40:06.847101 25772 tablet_replica.cc:330] T 4518087061c34feeb58acf1c7b9925c3 P 902217bf42594df99d3b3e5a7a34ab84: stopping tablet replica
I20260925 22:40:06.847246 25771 raft_consensus.cc:2258] T 4518087061c34feeb58acf1c7b9925c3 P 1fb59d2d608448ee8e548f9c3e522482 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:40:06.847571 25772 raft_consensus.cc:2229] T 4518087061c34feeb58acf1c7b9925c3 P 902217bf42594df99d3b3e5a7a34ab84 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:40:06.848142 25772 raft_consensus.cc:2258] T 4518087061c34feeb58acf1c7b9925c3 P 902217bf42594df99d3b3e5a7a34ab84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:40:06.849403 25771 ts_tablet_manager.cc:1905] T 4518087061c34feeb58acf1c7b9925c3 P 1fb59d2d608448ee8e548f9c3e522482: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:40:06.849684 25772 ts_tablet_manager.cc:1905] T 4518087061c34feeb58acf1c7b9925c3 P 902217bf42594df99d3b3e5a7a34ab84: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:40:06.852617 25772 ts_tablet_manager.cc:1918] T 4518087061c34feeb58acf1c7b9925c3 P 902217bf42594df99d3b3e5a7a34ab84: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:40:06.852770 25772 log.cc:1196] T 4518087061c34feeb58acf1c7b9925c3 P 902217bf42594df99d3b3e5a7a34ab84: Deleting WAL directory at /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/minicluster-data/ts-1/wal/wals/4518087061c34feeb58acf1c7b9925c3
I20260925 22:40:06.853186 25772 ts_tablet_manager.cc:1939] T 4518087061c34feeb58acf1c7b9925c3 P 902217bf42594df99d3b3e5a7a34ab84: Deleting consensus metadata
I20260925 22:40:06.853350 25771 ts_tablet_manager.cc:1918] T 4518087061c34feeb58acf1c7b9925c3 P 1fb59d2d608448ee8e548f9c3e522482: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:40:06.853473 25771 log.cc:1196] T 4518087061c34feeb58acf1c7b9925c3 P 1fb59d2d608448ee8e548f9c3e522482: Deleting WAL directory at /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790375752797504-7131-0/minicluster-data/ts-2/wal/wals/4518087061c34feeb58acf1c7b9925c3
I20260925 22:40:06.853921 25771 ts_tablet_manager.cc:1939] T 4518087061c34feeb58acf1c7b9925c3 P 1fb59d2d608448ee8e548f9c3e522482: Deleting consensus metadata
I20260925 22:40:06.855003 25255 catalog_manager.cc:4841] TS 902217bf42594df99d3b3e5a7a34ab84 (127.6.246.194:45211): tablet 4518087061c34feeb58acf1c7b9925c3 (table test-table [id=02144f290c924357ab0dbe66462cef4b]) successfully deleted
I20260925 22:40:06.855206 25257 catalog_manager.cc:4841] TS 1fb59d2d608448ee8e548f9c3e522482 (127.6.246.195:38759): tablet 4518087061c34feeb58acf1c7b9925c3 (table test-table [id=02144f290c924357ab0dbe66462cef4b]) successfully deleted
W20260925 22:40:06.855257 25254 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.246.193:35911 (error 108)
I20260925 22:40:06.856333 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 25466
I20260925 22:40:06.856614 25741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:06.856139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.193:35911 (local address 127.0.0.1:56946)
0925 22:40:06.856339 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:06.856440 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.193:35911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260925 22:40:06.858219 25254 catalog_manager.cc:4584] TS d44e1943717546bc866473b91e63a64f (127.6.246.193:35911): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4518087061c34feeb58acf1c7b9925c3: Network error: Client connection negotiation failed: client connection to 127.6.246.193:35911: connect: Connection refused (error 111)
I20260925 22:40:06.863296 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 25600
I20260925 22:40:06.870362 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 25238
2026-09-25T22:40:06Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2449 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-25T22:40:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:40:06Z Disabled control of system clock
I20260925 22:40:06.905514 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790375752797504-7131-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790375752797504-7131-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790375752797504-7131-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:41969
--webserver_interface=127.6.246.254
--webserver_port=0
--builtin_ntp_servers=127.6.246.212:45897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.246.254:41969
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 22:40:07.039274 25777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:40:07.039609 25777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:40:07.039650 25777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:40:07.044174 25777 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:40:07.044258 25777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:40:07.044273 25777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:40:07.044292 25777 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:40:07.044308 25777 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:40:07.049676 25777 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:45897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790375752797504-7131-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790375752797504-7131-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.246.254:41969
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:41969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.246.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25777
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790375752797504-7131-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:40:07.051069 25777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:40:07.052805 25777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:40:07.062950 25782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:07.062870 25783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:40:07.062934 25777 server_base.cc:1034] running on GCE node
W20260925 22:40:07.062870 25785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:40:07.063930 25777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:40:07.065373 25777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:40:07.066579 25777 hybrid_clock.cc:648] HybridClock initialized: now 1790376007066556 us; error 53 us; skew 500 ppm
I20260925 22:40:07.068929 25777 webserver.cc:460] Webserver started at http://127.6.246.254:42923/ using document root <none> and password file <none>
I20260925 22:40:07.069639 25777 fs_manager.cc:359] Metadata directory not provided
I20260925 22:40:07.069744 25777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:40:07.070022 25777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:40:07.072525 25777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790375752797504-7131-0/minicluster-data/master-0/data/instance:
uuid: "b91e5a65d7324c08ad57d3cfeb01353a"
format_stamp: "Formatted at 2026-09-25 22:40:07 on dist-test-slave-wjdr"
server_key: "15152c6f0c4af3e2a1b66be337cc7736"
server_key_iv: "51dbe21553f1098647c53703ba9191d9"
server_key_version: "encryptionkey@0"
I20260925 22:40:07.073210 25777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790375752797504-7131-0/minicluster-data/master-0/wal/instance:
uuid: "b91e5a65d7324c08ad57d3cfeb01353a"
format_stamp: "Formatted at 2026-09-25 22:40:07 on dist-test-slave-wjdr"
server_key: "15152c6f0c4af3e2a1b66be337cc7736"
server_key_iv: "51dbe21553f1098647c53703ba9191d9"
server_key_version: "encryptionkey@0"
I20260925 22:40:07.077625 25777 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:40:07.080451 25791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:07.082057 25777 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 22:40:07.082237 25777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790375752797504-7131-0/minicluster-data/master-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790375752797504-7131-0/minicluster-data/master-0/wal
uuid: "b91e5a65d7324c08ad57d3cfeb01353a"
format_stamp: "Formatted at 2026-09-25 22:40:07 on dist-test-slave-wjdr"
server_key: "15152c6f0c4af3e2a1b66be337cc7736"
server_key_iv: "51dbe21553f1098647c53703ba9191d9"
server_key_version: "encryptionkey@0"
I20260925 22:40:07.082369 25777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790375752797504-7131-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790375752797504-7131-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790375752797504-7131-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:40:07.096779 25777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:40:07.097620 25777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:40:07.097851 25777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:40:07.106120 25843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.254:41969 every 8 connection(s)
I20260925 22:40:07.106098 25777 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.254:41969
I20260925 22:40:07.107345 25777 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
I20260925 22:40:07.111218 25844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:40:07.113454 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 25777
I20260925 22:40:07.113626 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790375752797504-7131-0/minicluster-data/master-0/wal/instance
I20260925 22:40:07.113974 7131 external_mini_cluster.cc:1428] Setting key 3f3f06452660d9c88b9c41c91de65d1c
I20260925 22:40:07.118626 25844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b91e5a65d7324c08ad57d3cfeb01353a: Bootstrap starting.
I20260925 22:40:07.121723 25844 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b91e5a65d7324c08ad57d3cfeb01353a: Neither blocks nor log segments found. Creating new log.
I20260925 22:40:07.122826 25844 log.cc:824] T 00000000000000000000000000000000 P b91e5a65d7324c08ad57d3cfeb01353a: Log is configured to *not* fsync() on all Append() calls
I20260925 22:40:07.125110 25847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:07.115789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59918 (local address 127.6.246.254:41969)
0925 22:40:07.116575 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:07.116593 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:07.116636 (+ 43us) server_negotiation.cc:407] Connection header received
0925 22:40:07.117431 (+ 795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:07.117452 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:07.117924 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:07.118252 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:07.118688 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:07.119779 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:07.121077 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:07.121566 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:07.121696 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:07.122729 (+ 1033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:07.122735 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:07.122749 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:40:07.123296 (+ 547us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:07.123378 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:40:07.123408 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:07.123518 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:07.123679 (+ 161us) server_negotiation.cc:299] Negotiation successful
0925 22:40:07.123995 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":151,"threads_started":1}
I20260925 22:40:07.125470 25844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b91e5a65d7324c08ad57d3cfeb01353a: No bootstrap required, opened a new log
I20260925 22:40:07.129812 25844 raft_consensus.cc:348] T 00000000000000000000000000000000 P b91e5a65d7324c08ad57d3cfeb01353a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b91e5a65d7324c08ad57d3cfeb01353a" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 41969 } }
I20260925 22:40:07.130129 25844 raft_consensus.cc:374] T 00000000000000000000000000000000 P b91e5a65d7324c08ad57d3cfeb01353a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:40:07.130168 25844 raft_consensus.cc:729] T 00000000000000000000000000000000 P b91e5a65d7324c08ad57d3cfeb01353a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b91e5a65d7324c08ad57d3cfeb01353a, State: Initialized, Role: FOLLOWER
I20260925 22:40:07.130688 25844 consensus_queue.cc:260] T 00000000000000000000000000000000 P b91e5a65d7324c08ad57d3cfeb01353a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b91e5a65d7324c08ad57d3cfeb01353a" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 41969 } }
I20260925 22:40:07.130838 25844 raft_consensus.cc:388] T 00000000000000000000000000000000 P b91e5a65d7324c08ad57d3cfeb01353a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:40:07.130885 25844 raft_consensus.cc:482] T 00000000000000000000000000000000 P b91e5a65d7324c08ad57d3cfeb01353a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:40:07.131036 25844 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b91e5a65d7324c08ad57d3cfeb01353a [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:40:07.132447 25844 raft_consensus.cc:504] T 00000000000000000000000000000000 P b91e5a65d7324c08ad57d3cfeb01353a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b91e5a65d7324c08ad57d3cfeb01353a" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 41969 } }
I20260925 22:40:07.133198 25844 leader_election.cc:302] T 00000000000000000000000000000000 P b91e5a65d7324c08ad57d3cfeb01353a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b91e5a65d7324c08ad57d3cfeb01353a; no voters:
I20260925 22:40:07.133697 25844 leader_election.cc:288] T 00000000000000000000000000000000 P b91e5a65d7324c08ad57d3cfeb01353a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:40:07.133940 25849 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b91e5a65d7324c08ad57d3cfeb01353a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:40:07.134261 25849 raft_consensus.cc:686] T 00000000000000000000000000000000 P b91e5a65d7324c08ad57d3cfeb01353a [term 1 LEADER]: Becoming Leader. State: Replica: b91e5a65d7324c08ad57d3cfeb01353a, State: Running, Role: LEADER
I20260925 22:40:07.134678 25849 consensus_queue.cc:237] T 00000000000000000000000000000000 P b91e5a65d7324c08ad57d3cfeb01353a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b91e5a65d7324c08ad57d3cfeb01353a" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 41969 } }
I20260925 22:40:07.135640 25844 sys_catalog.cc:564] T 00000000000000000000000000000000 P b91e5a65d7324c08ad57d3cfeb01353a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:40:07.136801 25851 sys_catalog.cc:455] T 00000000000000000000000000000000 P b91e5a65d7324c08ad57d3cfeb01353a [sys.catalog]: SysCatalogTable state changed. Reason: New leader b91e5a65d7324c08ad57d3cfeb01353a. Latest consensus state: current_term: 1 leader_uuid: "b91e5a65d7324c08ad57d3cfeb01353a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b91e5a65d7324c08ad57d3cfeb01353a" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 41969 } } }
I20260925 22:40:07.137081 25851 sys_catalog.cc:458] T 00000000000000000000000000000000 P b91e5a65d7324c08ad57d3cfeb01353a [sys.catalog]: This master's current role is: LEADER
I20260925 22:40:07.137444 25850 sys_catalog.cc:455] T 00000000000000000000000000000000 P b91e5a65d7324c08ad57d3cfeb01353a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b91e5a65d7324c08ad57d3cfeb01353a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b91e5a65d7324c08ad57d3cfeb01353a" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 41969 } } }
I20260925 22:40:07.137573 25850 sys_catalog.cc:458] T 00000000000000000000000000000000 P b91e5a65d7324c08ad57d3cfeb01353a [sys.catalog]: This master's current role is: LEADER
I20260925 22:40:07.137535 25858 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:40:07.141891 25858 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:40:07.149799 25858 catalog_manager.cc:1347] Generated new cluster ID: 58fed5e9a7c845c1ae4d6d9a54998743
I20260925 22:40:07.149915 25858 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:40:07.163425 25858 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:40:07.165318 25858 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:40:07.175825 25858 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b91e5a65d7324c08ad57d3cfeb01353a: Generated new TSK 0
I20260925 22:40:07.176901 25858 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:40:07.188105 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790375752797504-7131-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790375752797504-7131-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790375752797504-7131-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.193:0
--local_ip_for_outbound_sockets=127.6.246.193
--webserver_interface=127.6.246.193
--webserver_port=0
--tserver_master_addrs=127.6.246.254:41969
--builtin_ntp_servers=127.6.246.212:45897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 22:40:07.322057 25868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:40:07.322366 25868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:40:07.322409 25868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:40:07.326660 25868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:40:07.326727 25868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:40:07.326788 25868 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.193
I20260925 22:40:07.331804 25868 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:45897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790375752797504-7131-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790375752797504-7131-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.246.193
--webserver_port=0
--tserver_master_addrs=127.6.246.254:41969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.193
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790375752797504-7131-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:40:07.333258 25868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:40:07.334949 25868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 22:40:07.346459 25868 server_base.cc:1034] running on GCE node
W20260925 22:40:07.346418 25874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:07.346441 25873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:07.346675 25876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:40:07.347112 25868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:40:07.347834 25868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:40:07.349002 25868 hybrid_clock.cc:648] HybridClock initialized: now 1790376007348982 us; error 30 us; skew 500 ppm
I20260925 22:40:07.351372 25868 webserver.cc:460] Webserver started at http://127.6.246.193:42693/ using document root <none> and password file <none>
I20260925 22:40:07.352051 25868 fs_manager.cc:359] Metadata directory not provided
I20260925 22:40:07.352118 25868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:40:07.352407 25868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:40:07.354882 25868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790375752797504-7131-0/minicluster-data/ts-0/data/instance:
uuid: "79b0f0a0e4c44723a95fd16e57511e72"
format_stamp: "Formatted at 2026-09-25 22:40:07 on dist-test-slave-wjdr"
server_key: "4db9c87dccca24667a6cb578413c4cad"
server_key_iv: "11d65502e4b7de5967b3228c72cb3917"
server_key_version: "encryptionkey@0"
I20260925 22:40:07.355441 25868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790375752797504-7131-0/minicluster-data/ts-0/wal/instance:
uuid: "79b0f0a0e4c44723a95fd16e57511e72"
format_stamp: "Formatted at 2026-09-25 22:40:07 on dist-test-slave-wjdr"
server_key: "4db9c87dccca24667a6cb578413c4cad"
server_key_iv: "11d65502e4b7de5967b3228c72cb3917"
server_key_version: "encryptionkey@0"
I20260925 22:40:07.359671 25868 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 22:40:07.362557 25882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:07.364089 25868 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:40:07.364336 25868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790375752797504-7131-0/minicluster-data/ts-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790375752797504-7131-0/minicluster-data/ts-0/wal
uuid: "79b0f0a0e4c44723a95fd16e57511e72"
format_stamp: "Formatted at 2026-09-25 22:40:07 on dist-test-slave-wjdr"
server_key: "4db9c87dccca24667a6cb578413c4cad"
server_key_iv: "11d65502e4b7de5967b3228c72cb3917"
server_key_version: "encryptionkey@0"
I20260925 22:40:07.364504 25868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790375752797504-7131-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790375752797504-7131-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790375752797504-7131-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:40:07.383520 25868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:40:07.384413 25868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:40:07.384696 25868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:40:07.385450 25868 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:40:07.386761 25868 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:40:07.386832 25868 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:07.386914 25868 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:40:07.386947 25868 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:07.397980 25868 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.193:32825
I20260925 22:40:07.398000 25995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.193:32825 every 8 connection(s)
I20260925 22:40:07.399057 25868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
I20260925 22:40:07.407318 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 25868
I20260925 22:40:07.407454 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790375752797504-7131-0/minicluster-data/ts-0/wal/instance
I20260925 22:40:07.407795 7131 external_mini_cluster.cc:1428] Setting key 6793e257e6e00e4c50469f526b166687
I20260925 22:40:07.409665 25847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:07.401249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.193:49975 (local address 127.6.246.254:41969)
0925 22:40:07.401427 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:07.401432 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:07.402528 (+ 1096us) server_negotiation.cc:407] Connection header received
0925 22:40:07.403978 (+ 1450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:07.403981 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:07.404033 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:07.404114 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:07.405110 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:07.405667 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:07.406827 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:07.407142 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:07.407287 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:07.408605 (+ 1318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:07.408609 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:07.408612 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:40:07.408625 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:07.408648 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:40:07.408658 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:07.408739 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:07.409428 (+ 689us) server_negotiation.cc:299] Negotiation successful
0925 22:40:07.409490 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 22:40:07.410089 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790375752797504-7131-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790375752797504-7131-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790375752797504-7131-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.194:0
--local_ip_for_outbound_sockets=127.6.246.194
--webserver_interface=127.6.246.194
--webserver_port=0
--tserver_master_addrs=127.6.246.254:41969
--builtin_ntp_servers=127.6.246.212:45897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 22:40:07.410672 25998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:07.401682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:41969 (local address 127.6.246.193:49975)
0925 22:40:07.402405 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:07.402444 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:07.403747 (+ 1303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:07.404276 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:07.404298 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:07.404379 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:40:07.404960 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:07.404973 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:07.405785 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:07.405787 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:07.406703 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:07.406712 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:07.407345 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:07.407349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:07.407553 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:07.408262 (+ 709us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:40:07.408270 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:40:07.408347 (+ 77us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:40:07.408432 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:40:07.409186 (+ 754us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:40:07.409194 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:40:07.409199 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 22:40:07.409395 (+ 196us) client_negotiation.cc:240] Negotiation successful
0925 22:40:07.409681 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":126,"threads_started":1}
I20260925 22:40:07.412050 25996 heartbeater.cc:347] Connected to a master server at 127.6.246.254:41969
I20260925 22:40:07.412331 25996 heartbeater.cc:464] Registering TS with master...
I20260925 22:40:07.412940 25996 heartbeater.cc:511] Master 127.6.246.254:41969 requested a full tablet report, sending...
I20260925 22:40:07.414616 25808 ts_manager.cc:194] Registered new tserver with Master: 79b0f0a0e4c44723a95fd16e57511e72 (127.6.246.193:32825)
I20260925 22:40:07.415843 25808 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.246.193:49975
W20260925 22:40:07.528649 25999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:40:07.528960 25999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:40:07.529038 25999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:40:07.533164 25999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:40:07.533257 25999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:40:07.533351 25999 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.194
I20260925 22:40:07.538539 25999 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:45897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790375752797504-7131-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790375752797504-7131-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.246.194
--webserver_port=0
--tserver_master_addrs=127.6.246.254:41969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25999
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.194
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790375752797504-7131-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:40:07.539680 25999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:40:07.541224 25999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:40:07.551056 26004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:07.551177 26005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:40:07.551096 25999 server_base.cc:1034] running on GCE node
W20260925 22:40:07.551066 26007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:40:07.551692 25999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:40:07.552343 25999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:40:07.553519 25999 hybrid_clock.cc:648] HybridClock initialized: now 1790376007553501 us; error 30 us; skew 500 ppm
I20260925 22:40:07.555532 25999 webserver.cc:460] Webserver started at http://127.6.246.194:33463/ using document root <none> and password file <none>
I20260925 22:40:07.556137 25999 fs_manager.cc:359] Metadata directory not provided
I20260925 22:40:07.556249 25999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:40:07.556500 25999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:40:07.558564 25999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790375752797504-7131-0/minicluster-data/ts-1/data/instance:
uuid: "4f8fa1e3fe514f18b4e41f9b25c1cd0e"
format_stamp: "Formatted at 2026-09-25 22:40:07 on dist-test-slave-wjdr"
server_key: "051917a512b7087c8e41d6c0e2757106"
server_key_iv: "51306df4d0928a52f6b9a76f798ddc20"
server_key_version: "encryptionkey@0"
I20260925 22:40:07.559060 25999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790375752797504-7131-0/minicluster-data/ts-1/wal/instance:
uuid: "4f8fa1e3fe514f18b4e41f9b25c1cd0e"
format_stamp: "Formatted at 2026-09-25 22:40:07 on dist-test-slave-wjdr"
server_key: "051917a512b7087c8e41d6c0e2757106"
server_key_iv: "51306df4d0928a52f6b9a76f798ddc20"
server_key_version: "encryptionkey@0"
I20260925 22:40:07.562742 25999 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 22:40:07.565258 26013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:07.566672 25999 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:40:07.566813 25999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790375752797504-7131-0/minicluster-data/ts-1/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790375752797504-7131-0/minicluster-data/ts-1/wal
uuid: "4f8fa1e3fe514f18b4e41f9b25c1cd0e"
format_stamp: "Formatted at 2026-09-25 22:40:07 on dist-test-slave-wjdr"
server_key: "051917a512b7087c8e41d6c0e2757106"
server_key_iv: "51306df4d0928a52f6b9a76f798ddc20"
server_key_version: "encryptionkey@0"
I20260925 22:40:07.566934 25999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790375752797504-7131-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790375752797504-7131-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790375752797504-7131-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:40:07.586517 25999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:40:07.587311 25999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:40:07.587517 25999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:40:07.588124 25999 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:40:07.589169 25999 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:40:07.589248 25999 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:07.589346 25999 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:40:07.589381 25999 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:07.598760 25999 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.194:34233
I20260925 22:40:07.598788 26126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.194:34233 every 8 connection(s)
I20260925 22:40:07.599635 25999 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
I20260925 22:40:07.607340 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 25999
I20260925 22:40:07.607463 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790375752797504-7131-0/minicluster-data/ts-1/wal/instance
I20260925 22:40:07.607839 7131 external_mini_cluster.cc:1428] Setting key 2f333d8f389d2256a46bfceac85f5b2c
I20260925 22:40:07.609123 25847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:07.601566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.194:38755 (local address 127.6.246.254:41969)
0925 22:40:07.601708 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:07.601712 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:07.602815 (+ 1103us) server_negotiation.cc:407] Connection header received
0925 22:40:07.604135 (+ 1320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:07.604138 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:07.604185 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:07.604269 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:07.605235 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:07.605800 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:07.606951 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:07.607156 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:07.607230 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:07.608601 (+ 1371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:07.608605 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:07.608608 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:40:07.608625 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:07.608645 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:40:07.608653 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:07.608745 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:07.608921 (+ 176us) server_negotiation.cc:299] Negotiation successful
0925 22:40:07.608983 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 22:40:07.609836 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790375752797504-7131-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790375752797504-7131-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790375752797504-7131-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.195:0
--local_ip_for_outbound_sockets=127.6.246.195
--webserver_interface=127.6.246.195
--webserver_port=0
--tserver_master_addrs=127.6.246.254:41969
--builtin_ntp_servers=127.6.246.212:45897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 22:40:07.609985 26129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:07.601986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:41969 (local address 127.6.246.194:38755)
0925 22:40:07.602631 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:07.602670 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:07.603940 (+ 1270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:07.604514 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:07.604521 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:07.604602 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:40:07.605097 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:07.605109 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:07.605943 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:07.605946 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:07.606833 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:07.606843 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:07.607414 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:07.607420 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:07.607621 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:07.608270 (+ 649us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:40:07.608272 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:40:07.608321 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:40:07.608381 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:40:07.608749 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:40:07.608760 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:40:07.608766 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 22:40:07.608909 (+ 143us) client_negotiation.cc:240] Negotiation successful
0925 22:40:07.609175 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":109,"threads_started":1}
I20260925 22:40:07.611117 26127 heartbeater.cc:347] Connected to a master server at 127.6.246.254:41969
I20260925 22:40:07.611337 26127 heartbeater.cc:464] Registering TS with master...
I20260925 22:40:07.611827 26127 heartbeater.cc:511] Master 127.6.246.254:41969 requested a full tablet report, sending...
I20260925 22:40:07.612830 25808 ts_manager.cc:194] Registered new tserver with Master: 4f8fa1e3fe514f18b4e41f9b25c1cd0e (127.6.246.194:34233)
I20260925 22:40:07.613401 25808 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.246.194:38755
W20260925 22:40:07.722107 26130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:40:07.722406 26130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:40:07.722479 26130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:40:07.726518 26130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:40:07.726595 26130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:40:07.726688 26130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.195
I20260925 22:40:07.731449 26130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:45897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790375752797504-7131-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790375752797504-7131-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.246.195
--webserver_port=0
--tserver_master_addrs=127.6.246.254:41969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.195
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790375752797504-7131-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:40:07.732570 26130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:40:07.733930 26130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:40:07.743680 26136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:07.743717 26135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:07.743703 26138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:40:07.744310 26130 server_base.cc:1034] running on GCE node
I20260925 22:40:07.744735 26130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:40:07.745301 26130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:40:07.746467 26130 hybrid_clock.cc:648] HybridClock initialized: now 1790376007746438 us; error 40 us; skew 500 ppm
I20260925 22:40:07.748366 26130 webserver.cc:460] Webserver started at http://127.6.246.195:39653/ using document root <none> and password file <none>
I20260925 22:40:07.748982 26130 fs_manager.cc:359] Metadata directory not provided
I20260925 22:40:07.749053 26130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:40:07.749275 26130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:40:07.751320 26130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790375752797504-7131-0/minicluster-data/ts-2/data/instance:
uuid: "d3dcf4afb5d848f596b8443be0aa6ddd"
format_stamp: "Formatted at 2026-09-25 22:40:07 on dist-test-slave-wjdr"
server_key: "5e152a9f17bbfda03034a4aa6491cf65"
server_key_iv: "a4d3705435902ff8734eca9be332474c"
server_key_version: "encryptionkey@0"
I20260925 22:40:07.751885 26130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790375752797504-7131-0/minicluster-data/ts-2/wal/instance:
uuid: "d3dcf4afb5d848f596b8443be0aa6ddd"
format_stamp: "Formatted at 2026-09-25 22:40:07 on dist-test-slave-wjdr"
server_key: "5e152a9f17bbfda03034a4aa6491cf65"
server_key_iv: "a4d3705435902ff8734eca9be332474c"
server_key_version: "encryptionkey@0"
I20260925 22:40:07.755640 26130 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 22:40:07.758188 26144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:07.759423 26130 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 22:40:07.759553 26130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790375752797504-7131-0/minicluster-data/ts-2/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790375752797504-7131-0/minicluster-data/ts-2/wal
uuid: "d3dcf4afb5d848f596b8443be0aa6ddd"
format_stamp: "Formatted at 2026-09-25 22:40:07 on dist-test-slave-wjdr"
server_key: "5e152a9f17bbfda03034a4aa6491cf65"
server_key_iv: "a4d3705435902ff8734eca9be332474c"
server_key_version: "encryptionkey@0"
I20260925 22:40:07.759690 26130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790375752797504-7131-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790375752797504-7131-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790375752797504-7131-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:40:07.777910 26130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:40:07.778632 26130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:40:07.778856 26130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:40:07.779465 26130 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:40:07.780421 26130 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:40:07.780483 26130 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:07.780558 26130 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:40:07.780601 26130 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:07.789716 26130 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.195:39073
I20260925 22:40:07.789734 26257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.195:39073 every 8 connection(s)
I20260925 22:40:07.790652 26130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
I20260925 22:40:07.796581 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 26130
I20260925 22:40:07.796700 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790375752797504-7131-0/minicluster-data/ts-2/wal/instance
I20260925 22:40:07.796967 7131 external_mini_cluster.cc:1428] Setting key 743f00b53d91d78a1a1e8e804ebbe54f
I20260925 22:40:07.799896 25847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:07.792390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.195:40699 (local address 127.6.246.254:41969)
0925 22:40:07.792558 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:07.792562 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:07.793561 (+ 999us) server_negotiation.cc:407] Connection header received
0925 22:40:07.794947 (+ 1386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:07.794950 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:07.795011 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:07.795120 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:07.795974 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:07.796523 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:07.797876 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:07.798167 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:07.798286 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:07.799401 (+ 1115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:07.799405 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:07.799408 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:40:07.799419 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:07.799440 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:40:07.799449 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:07.799532 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:07.799696 (+ 164us) server_negotiation.cc:299] Negotiation successful
0925 22:40:07.799755 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 22:40:07.800715 26260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:07.792780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:41969 (local address 127.6.246.195:40699)
0925 22:40:07.793407 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:07.793445 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:07.794731 (+ 1286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:07.795283 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:07.795290 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:07.795375 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:40:07.795818 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:07.795829 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:07.796660 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:07.796663 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:07.797754 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:07.797767 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:07.798327 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:07.798334 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:07.798514 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:07.799165 (+ 651us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:40:07.799167 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:40:07.799203 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:40:07.799260 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:40:07.799546 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:40:07.799556 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:40:07.799561 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 22:40:07.799682 (+ 121us) client_negotiation.cc:240] Negotiation successful
0925 22:40:07.799953 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":104,"threads_started":1}
I20260925 22:40:07.801759 26258 heartbeater.cc:347] Connected to a master server at 127.6.246.254:41969
I20260925 22:40:07.801990 26258 heartbeater.cc:464] Registering TS with master...
I20260925 22:40:07.802532 26258 heartbeater.cc:511] Master 127.6.246.254:41969 requested a full tablet report, sending...
I20260925 22:40:07.803524 25808 ts_manager.cc:194] Registered new tserver with Master: d3dcf4afb5d848f596b8443be0aa6ddd (127.6.246.195:39073)
I20260925 22:40:07.804054 25808 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.246.195:40699
I20260925 22:40:07.810741 7131 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:40:07.816283 25847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:07.812825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35762 (local address 127.6.246.254:41969)
0925 22:40:07.812957 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:07.812960 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:07.813079 (+ 119us) server_negotiation.cc:407] Connection header received
0925 22:40:07.813163 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:07.813165 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:07.813202 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:07.813267 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:07.813519 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:07.814022 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:07.814816 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:07.815017 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:07.815091 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:07.815843 (+ 752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:07.815847 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:07.815850 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:40:07.815862 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:07.815881 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:40:07.815889 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:07.815989 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:07.816137 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 22:40:07.816171 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 22:40:07.819182 25808 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35762:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:40:07.822397 25808 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:40:07.836655 26274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:07.831716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50334 (local address 127.6.246.193:32825)
0925 22:40:07.832070 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:07.832082 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:07.832104 (+ 22us) server_negotiation.cc:407] Connection header received
0925 22:40:07.832180 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:07.832185 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:07.832362 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:07.832524 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:40:07.832739 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:07.833815 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:07.835241 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:07.835946 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:07.836057 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:07.836259 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:07.836388 (+ 129us) server_negotiation.cc:299] Negotiation successful
0925 22:40:07.836516 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":87,"threads_started":1}
I20260925 22:40:07.836633 26269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:07.830692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.193:32825 (local address 127.0.0.1:50334)
0925 22:40:07.831310 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:07.831325 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:07.831409 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:07.832507 (+ 1098us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:07.832510 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:07.832529 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:40:07.832606 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:07.832612 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:07.833948 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:07.833951 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:07.835089 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:07.835101 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:07.836079 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:07.836085 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:07.836200 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:07.836236 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 22:40:07.836343 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 22:40:07.836480 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"mutex_wait_us":34,"thread_start_us":85,"threads_started":1}
I20260925 22:40:07.837003 26271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:07.830800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49074 (local address 127.6.246.194:34233)
0925 22:40:07.831267 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:07.831280 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:07.831304 (+ 24us) server_negotiation.cc:407] Connection header received
0925 22:40:07.831535 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:07.831541 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:07.831741 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:07.831899 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:40:07.832368 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:07.833336 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:07.835130 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:07.835996 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:07.836122 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:07.836297 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:07.836708 (+ 411us) server_negotiation.cc:299] Negotiation successful
0925 22:40:07.836806 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":162,"threads_started":1}
I20260925 22:40:07.837240 26273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:07.831675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59836 (local address 127.6.246.195:39073)
0925 22:40:07.832070 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:07.832082 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:07.832303 (+ 221us) server_negotiation.cc:407] Connection header received
0925 22:40:07.832549 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:07.832554 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:07.832723 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:07.832841 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:40:07.833246 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:07.834153 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:07.835541 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:07.836324 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:07.836421 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:07.836626 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:07.836976 (+ 350us) server_negotiation.cc:299] Negotiation successful
0925 22:40:07.837080 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":97,"threads_started":1}
I20260925 22:40:07.837280 26270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:07.830653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.194:34233 (local address 127.0.0.1:49074)
0925 22:40:07.831130 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:07.831182 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:07.831329 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:07.832067 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:07.832072 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:07.832092 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:40:07.832223 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:07.832243 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:07.833459 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:07.833462 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:07.835004 (+ 1542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:07.835012 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:07.836342 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:07.836348 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:07.836443 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:07.836489 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 22:40:07.836557 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 22:40:07.836626 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"mutex_wait_us":236,"thread_start_us":82,"threads_started":1}
I20260925 22:40:07.837669 26272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:07.831338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.195:39073 (local address 127.0.0.1:59836)
0925 22:40:07.832216 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:07.832229 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:07.832363 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:07.833011 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:07.833014 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:07.833031 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:40:07.833113 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:07.833119 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:07.834278 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:07.834281 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:07.835414 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:07.835422 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:07.836698 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:07.836701 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:07.836792 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:07.836812 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 22:40:07.837462 (+ 650us) client_negotiation.cc:240] Negotiation successful
0925 22:40:07.837538 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":818,"mutex_wait_us":64,"thread_start_us":87,"threads_started":1}
I20260925 22:40:07.840041 25930 tablet_service.cc:1467] Processing CreateTablet for tablet 00ac0ffad81f49beaeb5ba93ac4ef1c9 (DEFAULT_TABLE table=test-table [id=25f3bf91f8be4b2d8aef21f1bee3281f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:40:07.840085 26061 tablet_service.cc:1467] Processing CreateTablet for tablet 00ac0ffad81f49beaeb5ba93ac4ef1c9 (DEFAULT_TABLE table=test-table [id=25f3bf91f8be4b2d8aef21f1bee3281f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:40:07.840437 26192 tablet_service.cc:1467] Processing CreateTablet for tablet 00ac0ffad81f49beaeb5ba93ac4ef1c9 (DEFAULT_TABLE table=test-table [id=25f3bf91f8be4b2d8aef21f1bee3281f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:40:07.841128 25930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00ac0ffad81f49beaeb5ba93ac4ef1c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:40:07.841131 26061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00ac0ffad81f49beaeb5ba93ac4ef1c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:40:07.841308 26192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00ac0ffad81f49beaeb5ba93ac4ef1c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:40:07.847714 26275 tablet_bootstrap.cc:492] T 00ac0ffad81f49beaeb5ba93ac4ef1c9 P 79b0f0a0e4c44723a95fd16e57511e72: Bootstrap starting.
I20260925 22:40:07.848802 26277 tablet_bootstrap.cc:492] T 00ac0ffad81f49beaeb5ba93ac4ef1c9 P 4f8fa1e3fe514f18b4e41f9b25c1cd0e: Bootstrap starting.
I20260925 22:40:07.849212 26276 tablet_bootstrap.cc:492] T 00ac0ffad81f49beaeb5ba93ac4ef1c9 P d3dcf4afb5d848f596b8443be0aa6ddd: Bootstrap starting.
I20260925 22:40:07.850557 26275 tablet_bootstrap.cc:654] T 00ac0ffad81f49beaeb5ba93ac4ef1c9 P 79b0f0a0e4c44723a95fd16e57511e72: Neither blocks nor log segments found. Creating new log.
I20260925 22:40:07.850587 26277 tablet_bootstrap.cc:654] T 00ac0ffad81f49beaeb5ba93ac4ef1c9 P 4f8fa1e3fe514f18b4e41f9b25c1cd0e: Neither blocks nor log segments found. Creating new log.
I20260925 22:40:07.851212 26276 tablet_bootstrap.cc:654] T 00ac0ffad81f49beaeb5ba93ac4ef1c9 P d3dcf4afb5d848f596b8443be0aa6ddd: Neither blocks nor log segments found. Creating new log.
I20260925 22:40:07.851455 26277 log.cc:824] T 00ac0ffad81f49beaeb5ba93ac4ef1c9 P 4f8fa1e3fe514f18b4e41f9b25c1cd0e: Log is configured to *not* fsync() on all Append() calls
I20260925 22:40:07.851444 26275 log.cc:824] T 00ac0ffad81f49beaeb5ba93ac4ef1c9 P 79b0f0a0e4c44723a95fd16e57511e72: Log is configured to *not* fsync() on all Append() calls
I20260925 22:40:07.851965 26276 log.cc:824] T 00ac0ffad81f49beaeb5ba93ac4ef1c9 P d3dcf4afb5d848f596b8443be0aa6ddd: Log is configured to *not* fsync() on all Append() calls
I20260925 22:40:07.853881 26276 tablet_bootstrap.cc:492] T 00ac0ffad81f49beaeb5ba93ac4ef1c9 P d3dcf4afb5d848f596b8443be0aa6ddd: No bootstrap required, opened a new log
I20260925 22:40:07.853881 26275 tablet_bootstrap.cc:492] T 00ac0ffad81f49beaeb5ba93ac4ef1c9 P 79b0f0a0e4c44723a95fd16e57511e72: No bootstrap required, opened a new log
I20260925 22:40:07.854079 26277 tablet_bootstrap.cc:492] T 00ac0ffad81f49beaeb5ba93ac4ef1c9 P 4f8fa1e3fe514f18b4e41f9b25c1cd0e: No bootstrap required, opened a new log
I20260925 22:40:07.854090 26275 ts_tablet_manager.cc:1397] T 00ac0ffad81f49beaeb5ba93ac4ef1c9 P 79b0f0a0e4c44723a95fd16e57511e72: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260925 22:40:07.854090 26276 ts_tablet_manager.cc:1397] T 00ac0ffad81f49beaeb5ba93ac4ef1c9 P d3dcf4afb5d848f596b8443be0aa6ddd: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 22:40:07.854291 26277 ts_tablet_manager.cc:1397] T 00ac0ffad81f49beaeb5ba93ac4ef1c9 P 4f8fa1e3fe514f18b4e41f9b25c1cd0e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 22:40:07.857447 26277 raft_consensus.cc:348] T 00ac0ffad81f49beaeb5ba93ac4ef1c9 P 4f8fa1e3fe514f18b4e41f9b25c1cd0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8fa1e3fe514f18b4e41f9b25c1cd0e" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 34233 } } peers { permanent_uuid: "79b0f0a0e4c44723a95fd16e57511e72" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 32825 } } peers { permanent_uuid: "d3dcf4afb5d848f596b8443be0aa6ddd" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 39073 } }
I20260925 22:40:07.857704 26277 raft_consensus.cc:374] T 00ac0ffad81f49beaeb5ba93ac4ef1c9 P 4f8fa1e3fe514f18b4e41f9b25c1cd0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:40:07.857757 26277 raft_consensus.cc:729] T 00ac0ffad81f49beaeb5ba93ac4ef1c9 P 4f8fa1e3fe514f18b4e41f9b25c1cd0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f8fa1e3fe514f18b4e41f9b25c1cd0e, State: Initialized, Role: FOLLOWER
I20260925 22:40:07.858217 26277 consensus_queue.cc:260] T 00ac0ffad81f49beaeb5ba93ac4ef1c9 P 4f8fa1e3fe514f18b4e41f9b25c1cd0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8fa1e3fe514f18b4e41f9b25c1cd0e" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 34233 } } peers { permanent_uuid: "79b0f0a0e4c44723a95fd16e57511e72" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 32825 } } peers { permanent_uuid: "d3dcf4afb5d848f596b8443be0aa6ddd" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 39073 } }
I20260925 22:40:07.858340 26276 raft_consensus.cc:348] T 00ac0ffad81f49beaeb5ba93ac4ef1c9 P d3dcf4afb5d848f596b8443be0aa6ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8fa1e3fe514f18b4e41f9b25c1cd0e" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 34233 } } peers { permanent_uuid: "79b0f0a0e4c44723a95fd16e57511e72" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 32825 } } peers { permanent_uuid: "d3dcf4afb5d848f596b8443be0aa6ddd" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 39073 } }
I20260925 22:40:07.858340 26275 raft_consensus.cc:348] T 00ac0ffad81f49beaeb5ba93ac4ef1c9 P 79b0f0a0e4c44723a95fd16e57511e72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8fa1e3fe514f18b4e41f9b25c1cd0e" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 34233 } } peers { permanent_uuid: "79b0f0a0e4c44723a95fd16e57511e72" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 32825 } } peers { permanent_uuid: "d3dcf4afb5d848f596b8443be0aa6ddd" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 39073 } }
I20260925 22:40:07.858582 26276 raft_consensus.cc:374] T 00ac0ffad81f49beaeb5ba93ac4ef1c9 P d3dcf4afb5d848f596b8443be0aa6ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:40:07.858582 26275 raft_consensus.cc:374] T 00ac0ffad81f49beaeb5ba93ac4ef1c9 P 79b0f0a0e4c44723a95fd16e57511e72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:40:07.858639 26275 raft_consensus.cc:729] T 00ac0ffad81f49beaeb5ba93ac4ef1c9 P 79b0f0a0e4c44723a95fd16e57511e72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79b0f0a0e4c44723a95fd16e57511e72, State: Initialized, Role: FOLLOWER
I20260925 22:40:07.858639 26276 raft_consensus.cc:729] T 00ac0ffad81f49beaeb5ba93ac4ef1c9 P d3dcf4afb5d848f596b8443be0aa6ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3dcf4afb5d848f596b8443be0aa6ddd, State: Initialized, Role: FOLLOWER
I20260925 22:40:07.859035 26127 heartbeater.cc:503] Master 127.6.246.254:41969 was elected leader, sending a full tablet report...
I20260925 22:40:07.859061 26276 consensus_queue.cc:260] T 00ac0ffad81f49beaeb5ba93ac4ef1c9 P d3dcf4afb5d848f596b8443be0aa6ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8fa1e3fe514f18b4e41f9b25c1cd0e" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 34233 } } peers { permanent_uuid: "79b0f0a0e4c44723a95fd16e57511e72" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 32825 } } peers { permanent_uuid: "d3dcf4afb5d848f596b8443be0aa6ddd" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 39073 } }
I20260925 22:40:07.859097 26275 consensus_queue.cc:260] T 00ac0ffad81f49beaeb5ba93ac4ef1c9 P 79b0f0a0e4c44723a95fd16e57511e72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8fa1e3fe514f18b4e41f9b25c1cd0e" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 34233 } } peers { permanent_uuid: "79b0f0a0e4c44723a95fd16e57511e72" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 32825 } } peers { permanent_uuid: "d3dcf4afb5d848f596b8443be0aa6ddd" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 39073 } }
I20260925 22:40:07.860038 26276 ts_tablet_manager.cc:1428] T 00ac0ffad81f49beaeb5ba93ac4ef1c9 P d3dcf4afb5d848f596b8443be0aa6ddd: Time spent starting tablet: real 0.006s user 0.000s sys 0.007s
I20260925 22:40:07.860229 26275 ts_tablet_manager.cc:1428] T 00ac0ffad81f49beaeb5ba93ac4ef1c9 P 79b0f0a0e4c44723a95fd16e57511e72: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 22:40:07.860374 26258 heartbeater.cc:503] Master 127.6.246.254:41969 was elected leader, sending a full tablet report...
I20260925 22:40:07.860543 25996 heartbeater.cc:503] Master 127.6.246.254:41969 was elected leader, sending a full tablet report...
I20260925 22:40:07.861382 26277 ts_tablet_manager.cc:1428] T 00ac0ffad81f49beaeb5ba93ac4ef1c9 P 4f8fa1e3fe514f18b4e41f9b25c1cd0e: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260925 22:40:07.870334 26282 raft_consensus.cc:482] T 00ac0ffad81f49beaeb5ba93ac4ef1c9 P 79b0f0a0e4c44723a95fd16e57511e72 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:40:07.870563 26282 raft_consensus.cc:504] T 00ac0ffad81f49beaeb5ba93ac4ef1c9 P 79b0f0a0e4c44723a95fd16e57511e72 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8fa1e3fe514f18b4e41f9b25c1cd0e" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 34233 } } peers { permanent_uuid: "79b0f0a0e4c44723a95fd16e57511e72" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 32825 } } peers { permanent_uuid: "d3dcf4afb5d848f596b8443be0aa6ddd" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 39073 } }
I20260925 22:40:07.871673 26282 leader_election.cc:288] T 00ac0ffad81f49beaeb5ba93ac4ef1c9 P 79b0f0a0e4c44723a95fd16e57511e72 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f8fa1e3fe514f18b4e41f9b25c1cd0e (127.6.246.194:34233), d3dcf4afb5d848f596b8443be0aa6ddd (127.6.246.195:39073)
I20260925 22:40:07.875242 26271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:07.872071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.193:39967 (local address 127.6.246.194:34233)
0925 22:40:07.872196 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:07.872201 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:07.872218 (+ 17us) server_negotiation.cc:407] Connection header received
0925 22:40:07.872263 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:07.872265 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:07.872301 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:07.872396 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:40:07.872690 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:07.873202 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:07.874287 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:07.874681 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:07.874819 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:07.874868 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:07.875051 (+ 183us) server_negotiation.cc:299] Negotiation successful
0925 22:40:07.875117 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 22:40:07.875234 25998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:07.871881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.194:34233 (local address 127.6.246.193:39967)
0925 22:40:07.871979 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:07.871994 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:07.872076 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:07.872410 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:07.872413 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:07.872430 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:40:07.872572 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:07.872579 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:07.873323 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:07.873329 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:07.874152 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:07.874161 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:07.874813 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:07.874817 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:07.874914 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:07.874936 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 22:40:07.875037 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 22:40:07.875106 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":3}
I20260925 22:40:07.875671 26284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:07.871889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.195:39073 (local address 127.6.246.193:37035)
0925 22:40:07.872409 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:07.872420 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:07.872557 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:07.872816 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:07.872818 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:07.872833 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:40:07.872960 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:07.872966 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:07.873703 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:07.873708 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:07.874500 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:07.874508 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:07.875157 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:07.875161 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:07.875254 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:07.875280 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 22:40:07.875465 (+ 185us) client_negotiation.cc:240] Negotiation successful
0925 22:40:07.875528 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"mutex_wait_us":263,"thread_start_us":93,"threads_started":1}
I20260925 22:40:07.875701 26273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:07.871996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.193:37035 (local address 127.6.246.195:39073)
0925 22:40:07.872116 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:07.872122 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:07.872516 (+ 394us) server_negotiation.cc:407] Connection header received
0925 22:40:07.872662 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:07.872665 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:07.872707 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:07.872796 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:40:07.873082 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:07.873590 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:07.874626 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:07.875024 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:07.875157 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:07.875201 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:07.875502 (+ 301us) server_negotiation.cc:299] Negotiation successful
0925 22:40:07.875579 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 22:40:07.875909 26081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00ac0ffad81f49beaeb5ba93ac4ef1c9" candidate_uuid: "79b0f0a0e4c44723a95fd16e57511e72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f8fa1e3fe514f18b4e41f9b25c1cd0e" is_pre_election: true
I20260925 22:40:07.876243 26081 raft_consensus.cc:2454] T 00ac0ffad81f49beaeb5ba93ac4ef1c9 P 4f8fa1e3fe514f18b4e41f9b25c1cd0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79b0f0a0e4c44723a95fd16e57511e72 in term 0.
I20260925 22:40:07.876228 26212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00ac0ffad81f49beaeb5ba93ac4ef1c9" candidate_uuid: "79b0f0a0e4c44723a95fd16e57511e72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3dcf4afb5d848f596b8443be0aa6ddd" is_pre_election: true
I20260925 22:40:07.876492 26212 raft_consensus.cc:2454] T 00ac0ffad81f49beaeb5ba93ac4ef1c9 P d3dcf4afb5d848f596b8443be0aa6ddd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79b0f0a0e4c44723a95fd16e57511e72 in term 0.
I20260925 22:40:07.876760 25883 leader_election.cc:302] T 00ac0ffad81f49beaeb5ba93ac4ef1c9 P 79b0f0a0e4c44723a95fd16e57511e72 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f8fa1e3fe514f18b4e41f9b25c1cd0e, 79b0f0a0e4c44723a95fd16e57511e72; no voters:
I20260925 22:40:07.877027 26282 raft_consensus.cc:2781] T 00ac0ffad81f49beaeb5ba93ac4ef1c9 P 79b0f0a0e4c44723a95fd16e57511e72 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:40:07.877092 26282 raft_consensus.cc:482] T 00ac0ffad81f49beaeb5ba93ac4ef1c9 P 79b0f0a0e4c44723a95fd16e57511e72 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:40:07.877121 26282 raft_consensus.cc:3037] T 00ac0ffad81f49beaeb5ba93ac4ef1c9 P 79b0f0a0e4c44723a95fd16e57511e72 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:40:07.878108 26282 raft_consensus.cc:504] T 00ac0ffad81f49beaeb5ba93ac4ef1c9 P 79b0f0a0e4c44723a95fd16e57511e72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8fa1e3fe514f18b4e41f9b25c1cd0e" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 34233 } } peers { permanent_uuid: "79b0f0a0e4c44723a95fd16e57511e72" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 32825 } } peers { permanent_uuid: "d3dcf4afb5d848f596b8443be0aa6ddd" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 39073 } }
I20260925 22:40:07.878440 26282 leader_election.cc:288] T 00ac0ffad81f49beaeb5ba93ac4ef1c9 P 79b0f0a0e4c44723a95fd16e57511e72 [CANDIDATE]: Term 1 election: Requested vote from peers 4f8fa1e3fe514f18b4e41f9b25c1cd0e (127.6.246.194:34233), d3dcf4afb5d848f596b8443be0aa6ddd (127.6.246.195:39073)
I20260925 22:40:07.878782 26081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00ac0ffad81f49beaeb5ba93ac4ef1c9" candidate_uuid: "79b0f0a0e4c44723a95fd16e57511e72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f8fa1e3fe514f18b4e41f9b25c1cd0e"
I20260925 22:40:07.878791 26212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00ac0ffad81f49beaeb5ba93ac4ef1c9" candidate_uuid: "79b0f0a0e4c44723a95fd16e57511e72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3dcf4afb5d848f596b8443be0aa6ddd"
I20260925 22:40:07.878901 26081 raft_consensus.cc:3037] T 00ac0ffad81f49beaeb5ba93ac4ef1c9 P 4f8fa1e3fe514f18b4e41f9b25c1cd0e [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:40:07.878901 26212 raft_consensus.cc:3037] T 00ac0ffad81f49beaeb5ba93ac4ef1c9 P d3dcf4afb5d848f596b8443be0aa6ddd [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:40:07.880151 26212 raft_consensus.cc:2454] T 00ac0ffad81f49beaeb5ba93ac4ef1c9 P d3dcf4afb5d848f596b8443be0aa6ddd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79b0f0a0e4c44723a95fd16e57511e72 in term 1.
I20260925 22:40:07.880151 26081 raft_consensus.cc:2454] T 00ac0ffad81f49beaeb5ba93ac4ef1c9 P 4f8fa1e3fe514f18b4e41f9b25c1cd0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79b0f0a0e4c44723a95fd16e57511e72 in term 1.
I20260925 22:40:07.880504 25883 leader_election.cc:302] T 00ac0ffad81f49beaeb5ba93ac4ef1c9 P 79b0f0a0e4c44723a95fd16e57511e72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f8fa1e3fe514f18b4e41f9b25c1cd0e, 79b0f0a0e4c44723a95fd16e57511e72; no voters:
I20260925 22:40:07.880682 26282 raft_consensus.cc:2781] T 00ac0ffad81f49beaeb5ba93ac4ef1c9 P 79b0f0a0e4c44723a95fd16e57511e72 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:40:07.880888 26282 raft_consensus.cc:686] T 00ac0ffad81f49beaeb5ba93ac4ef1c9 P 79b0f0a0e4c44723a95fd16e57511e72 [term 1 LEADER]: Becoming Leader. State: Replica: 79b0f0a0e4c44723a95fd16e57511e72, State: Running, Role: LEADER
I20260925 22:40:07.881235 26282 consensus_queue.cc:237] T 00ac0ffad81f49beaeb5ba93ac4ef1c9 P 79b0f0a0e4c44723a95fd16e57511e72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8fa1e3fe514f18b4e41f9b25c1cd0e" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 34233 } } peers { permanent_uuid: "79b0f0a0e4c44723a95fd16e57511e72" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 32825 } } peers { permanent_uuid: "d3dcf4afb5d848f596b8443be0aa6ddd" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 39073 } }
I20260925 22:40:07.884253 25808 catalog_manager.cc:5495] T 00ac0ffad81f49beaeb5ba93ac4ef1c9 P 79b0f0a0e4c44723a95fd16e57511e72 reported cstate change: term changed from 0 to 1, leader changed from <none> to 79b0f0a0e4c44723a95fd16e57511e72 (127.6.246.193). New cstate: current_term: 1 leader_uuid: "79b0f0a0e4c44723a95fd16e57511e72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8fa1e3fe514f18b4e41f9b25c1cd0e" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 34233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79b0f0a0e4c44723a95fd16e57511e72" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 32825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3dcf4afb5d848f596b8443be0aa6ddd" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 39073 } health_report { overall_health: UNKNOWN } } }
I20260925 22:40:07.900164 26274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:07.896697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50348 (local address 127.6.246.193:32825)
0925 22:40:07.896863 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:07.896866 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:07.896880 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:40:07.896942 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:07.896944 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:07.897003 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:07.897097 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:40:07.897308 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:07.897827 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:07.898628 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:07.898867 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:07.898945 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:07.899273 (+ 328us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:40:07.899670 (+ 397us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:40:07.899820 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:07.899977 (+ 157us) server_negotiation.cc:299] Negotiation successful
0925 22:40:07.900009 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
W20260925 22:40:07.900357 25997 tablet.cc:2369] T 00ac0ffad81f49beaeb5ba93ac4ef1c9 P 79b0f0a0e4c44723a95fd16e57511e72: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:40:07.905933 26081 raft_consensus.cc:1261] T 00ac0ffad81f49beaeb5ba93ac4ef1c9 P 4f8fa1e3fe514f18b4e41f9b25c1cd0e [term 1 FOLLOWER]: Refusing update from remote peer 79b0f0a0e4c44723a95fd16e57511e72: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:40:07.905934 26212 raft_consensus.cc:1261] T 00ac0ffad81f49beaeb5ba93ac4ef1c9 P d3dcf4afb5d848f596b8443be0aa6ddd [term 1 FOLLOWER]: Refusing update from remote peer 79b0f0a0e4c44723a95fd16e57511e72: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:40:07.906646 26282 consensus_queue.cc:1035] T 00ac0ffad81f49beaeb5ba93ac4ef1c9 P 79b0f0a0e4c44723a95fd16e57511e72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f8fa1e3fe514f18b4e41f9b25c1cd0e" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 34233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:40:07.906840 26285 consensus_queue.cc:1035] T 00ac0ffad81f49beaeb5ba93ac4ef1c9 P 79b0f0a0e4c44723a95fd16e57511e72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3dcf4afb5d848f596b8443be0aa6ddd" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 39073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:40:07.913539 26288 mvcc.cc:205] Tried to move back new op lower bound from 7333380128375513088 to 7333380128284184576. Current Snapshot: MvccSnapshot[applied={T|T < 7333380128375513088}]
I20260925 22:40:07.914686 26290 mvcc.cc:205] Tried to move back new op lower bound from 7333380128375513088 to 7333380128284184576. Current Snapshot: MvccSnapshot[applied={T|T < 7333380128375513088}]
I20260925 22:40:07.914985 26289 mvcc.cc:205] Tried to move back new op lower bound from 7333380128375513088 to 7333380128284184576. Current Snapshot: MvccSnapshot[applied={T|T < 7333380128375513088}]
I20260925 22:40:07.923527 25808 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:40:07.923669 25808 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:40:07.926070 25808 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b91e5a65d7324c08ad57d3cfeb01353a: Sending DeleteTablet for 3 replicas of tablet 00ac0ffad81f49beaeb5ba93ac4ef1c9
I20260925 22:40:07.926890 26061 tablet_service.cc:1514] Processing DeleteTablet for tablet 00ac0ffad81f49beaeb5ba93ac4ef1c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:40:07 UTC) from {username='slave'} at 127.0.0.1:49074
I20260925 22:40:07.927110 25930 tablet_service.cc:1514] Processing DeleteTablet for tablet 00ac0ffad81f49beaeb5ba93ac4ef1c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:40:07 UTC) from {username='slave'} at 127.0.0.1:50334
I20260925 22:40:07.927356 26192 tablet_service.cc:1514] Processing DeleteTablet for tablet 00ac0ffad81f49beaeb5ba93ac4ef1c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:40:07 UTC) from {username='slave'} at 127.0.0.1:59836
I20260925 22:40:07.927587 26298 tablet_replica.cc:330] T 00ac0ffad81f49beaeb5ba93ac4ef1c9 P 4f8fa1e3fe514f18b4e41f9b25c1cd0e: stopping tablet replica
I20260925 22:40:07.927585 26299 tablet_replica.cc:330] T 00ac0ffad81f49beaeb5ba93ac4ef1c9 P 79b0f0a0e4c44723a95fd16e57511e72: stopping tablet replica
I20260925 22:40:07.927877 26299 raft_consensus.cc:2229] T 00ac0ffad81f49beaeb5ba93ac4ef1c9 P 79b0f0a0e4c44723a95fd16e57511e72 [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:40:07.927877 26298 raft_consensus.cc:2229] T 00ac0ffad81f49beaeb5ba93ac4ef1c9 P 4f8fa1e3fe514f18b4e41f9b25c1cd0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:40:07.928164 26298 raft_consensus.cc:2258] T 00ac0ffad81f49beaeb5ba93ac4ef1c9 P 4f8fa1e3fe514f18b4e41f9b25c1cd0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:40:07.928217 26299 raft_consensus.cc:2258] T 00ac0ffad81f49beaeb5ba93ac4ef1c9 P 79b0f0a0e4c44723a95fd16e57511e72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:40:07.928364 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 25868
I20260925 22:40:07.930194 26300 tablet_replica.cc:330] T 00ac0ffad81f49beaeb5ba93ac4ef1c9 P d3dcf4afb5d848f596b8443be0aa6ddd: stopping tablet replica
I20260925 22:40:07.930524 26300 raft_consensus.cc:2229] T 00ac0ffad81f49beaeb5ba93ac4ef1c9 P d3dcf4afb5d848f596b8443be0aa6ddd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:40:07.930827 26300 raft_consensus.cc:2258] T 00ac0ffad81f49beaeb5ba93ac4ef1c9 P d3dcf4afb5d848f596b8443be0aa6ddd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:40:07.931421 26298 ts_tablet_manager.cc:1905] T 00ac0ffad81f49beaeb5ba93ac4ef1c9 P 4f8fa1e3fe514f18b4e41f9b25c1cd0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:40:07.932130 26300 ts_tablet_manager.cc:1905] T 00ac0ffad81f49beaeb5ba93ac4ef1c9 P d3dcf4afb5d848f596b8443be0aa6ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:40:07.934350 26298 ts_tablet_manager.cc:1918] T 00ac0ffad81f49beaeb5ba93ac4ef1c9 P 4f8fa1e3fe514f18b4e41f9b25c1cd0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:40:07.934451 26298 log.cc:1196] T 00ac0ffad81f49beaeb5ba93ac4ef1c9 P 4f8fa1e3fe514f18b4e41f9b25c1cd0e: Deleting WAL directory at /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790375752797504-7131-0/minicluster-data/ts-1/wal/wals/00ac0ffad81f49beaeb5ba93ac4ef1c9
I20260925 22:40:07.934796 26298 ts_tablet_manager.cc:1939] T 00ac0ffad81f49beaeb5ba93ac4ef1c9 P 4f8fa1e3fe514f18b4e41f9b25c1cd0e: Deleting consensus metadata
W20260925 22:40:07.935062 25793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.246.193:32825 (error 108)
I20260925 22:40:07.935310 26300 ts_tablet_manager.cc:1918] T 00ac0ffad81f49beaeb5ba93ac4ef1c9 P d3dcf4afb5d848f596b8443be0aa6ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:40:07.935406 26300 log.cc:1196] T 00ac0ffad81f49beaeb5ba93ac4ef1c9 P d3dcf4afb5d848f596b8443be0aa6ddd: Deleting WAL directory at /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790375752797504-7131-0/minicluster-data/ts-2/wal/wals/00ac0ffad81f49beaeb5ba93ac4ef1c9
I20260925 22:40:07.935726 26300 ts_tablet_manager.cc:1939] T 00ac0ffad81f49beaeb5ba93ac4ef1c9 P d3dcf4afb5d848f596b8443be0aa6ddd: Deleting consensus metadata
I20260925 22:40:07.935899 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 25999
I20260925 22:40:07.936177 26272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:07.935904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.193:32825 (local address 127.0.0.1:50352)
0925 22:40:07.936034 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:07.936102 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.193:32825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260925 22:40:07.936554 25793 catalog_manager.cc:4584] TS 79b0f0a0e4c44723a95fd16e57511e72 (127.6.246.193:32825): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 00ac0ffad81f49beaeb5ba93ac4ef1c9: Network error: Client connection negotiation failed: client connection to 127.6.246.193:32825: connect: Connection refused (error 111)
I20260925 22:40:07.937723 25793 catalog_manager.cc:4841] TS d3dcf4afb5d848f596b8443be0aa6ddd (127.6.246.195:39073): tablet 00ac0ffad81f49beaeb5ba93ac4ef1c9 (table test-table [id=25f3bf91f8be4b2d8aef21f1bee3281f]) successfully deleted
I20260925 22:40:07.942493 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 26130
I20260925 22:40:07.943331 26272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:07.942607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.194:34233 (local address 127.0.0.1:49088)
0925 22:40:07.943167 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:07.943231 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.194:34233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":460}
I20260925 22:40:07.948725 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 25777
2026-09-25T22:40:07Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1074 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:40:08 dist-test-slave-wjdr krb5kdc[26304](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:40:08 dist-test-slave-wjdr krb5kdc[26304](info): set up 2 sockets
Sep 25 22:40:08 dist-test-slave-wjdr krb5kdc[26304](info): commencing operation
krb5kdc: starting...
W20260925 22:40:08.996531 7131 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Sep 25 22:40:09 dist-test-slave-wjdr krb5kdc[26304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:40:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:40:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.246.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab.
Entry for principal kudu/127.6.246.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab.
I20260925 22:40:09.124948 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:39837
--webserver_interface=127.6.246.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab
--principal=kudu/127.6.246.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:41757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.246.254:39837
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:40:09.235816 26317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:40:09.236086 26317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:40:09.236127 26317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:40:09.240222 26317 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:40:09.240271 26317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:40:09.240290 26317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:40:09.240309 26317 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:40:09.240324 26317 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:40:09.245666 26317 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:41757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.246.254:39837
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab
--principal=kudu/127.6.246.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:39837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.246.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26317
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:40:09.246688 26317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:40:09.247713 26317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:40:09.256136 26322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:09.256130 26323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:09.256130 26325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:40:09.256590 26317 server_base.cc:1034] running on GCE node
I20260925 22:40:09.257296 26317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:40:09.258373 26317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:40:09.259577 26317 hybrid_clock.cc:648] HybridClock initialized: now 1790376009259552 us; error 47 us; skew 500 ppm
Sep 25 22:40:09 dist-test-slave-wjdr krb5kdc[26304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376009, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:40:09.262507 26317 init.cc:377] Logged in from keytab as kudu/127.6.246.254@KRBTEST.COM (short username kudu)
I20260925 22:40:09.263517 26317 webserver.cc:460] Webserver started at http://127.6.246.254:37881/ using document root <none> and password file <none>
I20260925 22:40:09.264120 26317 fs_manager.cc:359] Metadata directory not provided
I20260925 22:40:09.264220 26317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:40:09.264433 26317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:40:09.266455 26317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/minicluster-data/master-0/data/instance:
uuid: "c6dbbadfc1be459895dff1ea035c74d5"
format_stamp: "Formatted at 2026-09-25 22:40:09 on dist-test-slave-wjdr"
server_key: "fe910b4f75e027f30fd2b89d025b06ea"
server_key_iv: "18988704edfa64239f8c4a2b4dcaa66f"
server_key_version: "encryptionkey@0"
I20260925 22:40:09.267078 26317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/minicluster-data/master-0/wal/instance:
uuid: "c6dbbadfc1be459895dff1ea035c74d5"
format_stamp: "Formatted at 2026-09-25 22:40:09 on dist-test-slave-wjdr"
server_key: "fe910b4f75e027f30fd2b89d025b06ea"
server_key_iv: "18988704edfa64239f8c4a2b4dcaa66f"
server_key_version: "encryptionkey@0"
I20260925 22:40:09.270992 26317 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:40:09.273427 26332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:09.274817 26317 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:40:09.274956 26317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/minicluster-data/master-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/minicluster-data/master-0/wal
uuid: "c6dbbadfc1be459895dff1ea035c74d5"
format_stamp: "Formatted at 2026-09-25 22:40:09 on dist-test-slave-wjdr"
server_key: "fe910b4f75e027f30fd2b89d025b06ea"
server_key_iv: "18988704edfa64239f8c4a2b4dcaa66f"
server_key_version: "encryptionkey@0"
I20260925 22:40:09.275071 26317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:40:09.289978 26317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:40:09.293038 26317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:40:09.293249 26317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:40:09.300608 26317 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.254:39837
I20260925 22:40:09.300647 26384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.254:39837 every 8 connection(s)
I20260925 22:40:09.301631 26317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
I20260925 22:40:09.304327 26385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:40:09.309660 26385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6dbbadfc1be459895dff1ea035c74d5: Bootstrap starting.
I20260925 22:40:09.311388 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 26317
I20260925 22:40:09.311542 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/minicluster-data/master-0/wal/instance
I20260925 22:40:09.311830 7131 external_mini_cluster.cc:1428] Setting key d4bb21655fca0dd925f892b728712cc0
I20260925 22:40:09.312191 26385 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6dbbadfc1be459895dff1ea035c74d5: Neither blocks nor log segments found. Creating new log.
I20260925 22:40:09.313084 26385 log.cc:824] T 00000000000000000000000000000000 P c6dbbadfc1be459895dff1ea035c74d5: Log is configured to *not* fsync() on all Append() calls
I20260925 22:40:09.314925 26385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6dbbadfc1be459895dff1ea035c74d5: No bootstrap required, opened a new log
I20260925 22:40:09.318307 26385 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6dbbadfc1be459895dff1ea035c74d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6dbbadfc1be459895dff1ea035c74d5" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 39837 } }
I20260925 22:40:09.318547 26385 raft_consensus.cc:374] T 00000000000000000000000000000000 P c6dbbadfc1be459895dff1ea035c74d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:40:09.318612 26385 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6dbbadfc1be459895dff1ea035c74d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6dbbadfc1be459895dff1ea035c74d5, State: Initialized, Role: FOLLOWER
I20260925 22:40:09.319083 26385 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6dbbadfc1be459895dff1ea035c74d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6dbbadfc1be459895dff1ea035c74d5" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 39837 } }
I20260925 22:40:09.319260 26385 raft_consensus.cc:388] T 00000000000000000000000000000000 P c6dbbadfc1be459895dff1ea035c74d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:40:09.319335 26385 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6dbbadfc1be459895dff1ea035c74d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:40:09.319447 26385 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6dbbadfc1be459895dff1ea035c74d5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:40:09.320534 26385 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6dbbadfc1be459895dff1ea035c74d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6dbbadfc1be459895dff1ea035c74d5" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 39837 } }
Sep 25 22:40:09 dist-test-slave-wjdr krb5kdc[26304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
I20260925 22:40:09.320887 26385 leader_election.cc:302] T 00000000000000000000000000000000 P c6dbbadfc1be459895dff1ea035c74d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6dbbadfc1be459895dff1ea035c74d5; no voters:
I20260925 22:40:09.321280 26385 leader_election.cc:288] T 00000000000000000000000000000000 P c6dbbadfc1be459895dff1ea035c74d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:40:09.321321 26390 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6dbbadfc1be459895dff1ea035c74d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:40:09.321497 26390 raft_consensus.cc:686] T 00000000000000000000000000000000 P c6dbbadfc1be459895dff1ea035c74d5 [term 1 LEADER]: Becoming Leader. State: Replica: c6dbbadfc1be459895dff1ea035c74d5, State: Running, Role: LEADER
I20260925 22:40:09.321827 26390 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6dbbadfc1be459895dff1ea035c74d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6dbbadfc1be459895dff1ea035c74d5" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 39837 } }
I20260925 22:40:09.322448 26385 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6dbbadfc1be459895dff1ea035c74d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:40:09.323441 26392 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6dbbadfc1be459895dff1ea035c74d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c6dbbadfc1be459895dff1ea035c74d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6dbbadfc1be459895dff1ea035c74d5" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 39837 } } }
I20260925 22:40:09.323577 26392 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6dbbadfc1be459895dff1ea035c74d5 [sys.catalog]: This master's current role is: LEADER
I20260925 22:40:09.323869 26391 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6dbbadfc1be459895dff1ea035c74d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6dbbadfc1be459895dff1ea035c74d5. Latest consensus state: current_term: 1 leader_uuid: "c6dbbadfc1be459895dff1ea035c74d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6dbbadfc1be459895dff1ea035c74d5" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 39837 } } }
I20260925 22:40:09.324026 26391 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6dbbadfc1be459895dff1ea035c74d5 [sys.catalog]: This master's current role is: LEADER
I20260925 22:40:09.326975 26388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:09.312998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59836 (local address 127.6.246.254:39837)
0925 22:40:09.313430 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:09.313458 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:09.313507 (+ 49us) server_negotiation.cc:407] Connection header received
0925 22:40:09.314265 (+ 758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:09.314281 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:09.314582 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:09.314915 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:09.315705 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:09.316752 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:09.317630 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:09.318083 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:09.318267 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:09.321558 (+ 3291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:09.321565 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:09.321582 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:09.321612 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:09.323713 (+ 2101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:09.324251 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:09.324257 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:09.324262 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:09.324426 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:09.324791 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:09.324794 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:09.324795 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:09.325443 (+ 648us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:09.325603 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:09.325949 (+ 346us) server_negotiation.cc:299] Negotiation successful
0925 22:40:09.326301 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":123,"threads_started":1}
W20260925 22:40:09.327792 26406 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c6dbbadfc1be459895dff1ea035c74d5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:40:09.327929 26406 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:40:09.328032 26399 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:40:09.328917 26399 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:40:09.334115 26399 catalog_manager.cc:1347] Generated new cluster ID: bd2322810ead4e5a9ce1358f44b615ed
I20260925 22:40:09.334173 26399 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:40:09.372229 26399 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:40:09.373621 26399 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:40:09.385823 26399 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c6dbbadfc1be459895dff1ea035c74d5: Generated new TSK 0
I20260925 22:40:09.386595 26399 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.246.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.193.keytab.
Entry for principal kudu/127.6.246.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.193.keytab.
I20260925 22:40:09.420339 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.193:0
--local_ip_for_outbound_sockets=127.6.246.193
--webserver_interface=127.6.246.193
--webserver_port=0
--tserver_master_addrs=127.6.246.254:39837
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.193.keytab
--principal=kudu/127.6.246.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:41757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:40:09.530475 26411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:40:09.530704 26411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:40:09.530745 26411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:40:09.534730 26411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:40:09.534782 26411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:40:09.534881 26411 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.193
I20260925 22:40:09.540047 26411 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:41757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.193.keytab
--principal=kudu/127.6.246.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.246.193
--webserver_port=0
--tserver_master_addrs=127.6.246.254:39837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26411
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.193
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:40:09.541056 26411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:40:09.542030 26411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:40:09.551326 26417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:09.551331 26419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:40:09.551530 26411 server_base.cc:1034] running on GCE node
W20260925 22:40:09.551352 26416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:40:09.552045 26411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:40:09.552711 26411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:40:09.553943 26411 hybrid_clock.cc:648] HybridClock initialized: now 1790376009553910 us; error 45 us; skew 500 ppm
Sep 25 22:40:09 dist-test-slave-wjdr krb5kdc[26304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376009, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:40:09.556763 26411 init.cc:377] Logged in from keytab as kudu/127.6.246.193@KRBTEST.COM (short username kudu)
I20260925 22:40:09.557542 26411 webserver.cc:460] Webserver started at http://127.6.246.193:36275/ using document root <none> and password file <none>
I20260925 22:40:09.558058 26411 fs_manager.cc:359] Metadata directory not provided
I20260925 22:40:09.558116 26411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:40:09.558274 26411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:40:09.560228 26411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/minicluster-data/ts-0/data/instance:
uuid: "5843a00aa3d74373a1ba83c86f80e02d"
format_stamp: "Formatted at 2026-09-25 22:40:09 on dist-test-slave-wjdr"
server_key: "79f8d1dcd062329e80167b1fb1628682"
server_key_iv: "390506b7e1455e5512cb3419ae3b96fc"
server_key_version: "encryptionkey@0"
I20260925 22:40:09.560921 26411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/minicluster-data/ts-0/wal/instance:
uuid: "5843a00aa3d74373a1ba83c86f80e02d"
format_stamp: "Formatted at 2026-09-25 22:40:09 on dist-test-slave-wjdr"
server_key: "79f8d1dcd062329e80167b1fb1628682"
server_key_iv: "390506b7e1455e5512cb3419ae3b96fc"
server_key_version: "encryptionkey@0"
I20260925 22:40:09.564445 26411 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:40:09.567094 26426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:09.568434 26411 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:40:09.568626 26411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/minicluster-data/ts-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/minicluster-data/ts-0/wal
uuid: "5843a00aa3d74373a1ba83c86f80e02d"
format_stamp: "Formatted at 2026-09-25 22:40:09 on dist-test-slave-wjdr"
server_key: "79f8d1dcd062329e80167b1fb1628682"
server_key_iv: "390506b7e1455e5512cb3419ae3b96fc"
server_key_version: "encryptionkey@0"
I20260925 22:40:09.568742 26411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:40:09.592188 26411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:40:09.595763 26411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:40:09.595995 26411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:40:09.596735 26411 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:40:09.597600 26411 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:40:09.597646 26411 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:09.597718 26411 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:40:09.597752 26411 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:09.606923 26411 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.193:38643
I20260925 22:40:09.606976 26539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.193:38643 every 8 connection(s)
I20260925 22:40:09.607887 26411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
I20260925 22:40:09.617610 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 26411
I20260925 22:40:09.617762 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/minicluster-data/ts-0/wal/instance
I20260925 22:40:09.618108 7131 external_mini_cluster.cc:1428] Setting key 53d2fbf6fa4818b4aa3c51359b48aca8
Sep 25 22:40:09 dist-test-slave-wjdr krb5kdc[26304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376009, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.193@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
I20260925 22:40:09.623618 26388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:09.609898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.193:39867 (local address 127.6.246.254:39837)
0925 22:40:09.610062 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:09.610066 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:09.610763 (+ 697us) server_negotiation.cc:407] Connection header received
0925 22:40:09.612263 (+ 1500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:09.612267 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:09.612333 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:09.612420 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:09.613749 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:09.614360 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:09.615619 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:09.615889 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:09.616022 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:09.618978 (+ 2956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:09.618982 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:09.618984 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:09.619015 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:09.620819 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:09.621569 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:09.621574 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:09.621576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:09.621692 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:09.622178 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:09.622183 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:09.622188 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:09.622480 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:09.622604 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:09.623154 (+ 550us) server_negotiation.cc:299] Negotiation successful
0925 22:40:09.623340 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 22:40:09.624428 26542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:09.610107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:39837 (local address 127.6.246.193:39867)
0925 22:40:09.610581 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:09.610624 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:09.612023 (+ 1399us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:09.612647 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:09.612653 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:09.613033 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:40:09.613560 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:09.613573 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:09.614508 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:09.614513 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:09.615473 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:09.615482 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:09.616065 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:09.616070 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:09.616255 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:09.616908 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:40:09.616925 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:40:09.618745 (+ 1820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:40:09.620945 (+ 2200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:09.620954 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:09.620975 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:09.621397 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:09.621844 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:09.621847 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:09.621848 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:09.622059 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:09.622599 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:40:09.622604 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:40:09.622867 (+ 263us) client_negotiation.cc:769] Sending connection context
0925 22:40:09.623033 (+ 166us) client_negotiation.cc:240] Negotiation successful
0925 22:40:09.623424 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":123,"threads_started":1}
I20260925 22:40:09.626005 26540 heartbeater.cc:347] Connected to a master server at 127.6.246.254:39837
I20260925 22:40:09.626261 26540 heartbeater.cc:464] Registering TS with master...
I20260925 22:40:09.626793 26540 heartbeater.cc:511] Master 127.6.246.254:39837 requested a full tablet report, sending...
I20260925 22:40:09.628419 26349 ts_manager.cc:194] Registered new tserver with Master: 5843a00aa3d74373a1ba83c86f80e02d (127.6.246.193:38643)
WARNING: no policy specified for kudu/127.6.246.194@KRBTEST.COM; defaulting to no policy
I20260925 22:40:09.629673 26349 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.246.193@KRBTEST.COM'} at 127.6.246.193:39867
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.194.keytab.
Entry for principal kudu/127.6.246.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.194.keytab.
I20260925 22:40:09.648145 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.194:0
--local_ip_for_outbound_sockets=127.6.246.194
--webserver_interface=127.6.246.194
--webserver_port=0
--tserver_master_addrs=127.6.246.254:39837
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.194.keytab
--principal=kudu/127.6.246.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:41757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:40:09.761114 26545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:40:09.761453 26545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:40:09.761502 26545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:40:09.765708 26545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:40:09.765775 26545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:40:09.765838 26545 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.194
I20260925 22:40:09.771215 26545 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:41757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.194.keytab
--principal=kudu/127.6.246.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.246.194
--webserver_port=0
--tserver_master_addrs=127.6.246.254:39837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26545
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.194
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:40:09.772367 26545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:40:09.773569 26545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:40:09.783720 26551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:09.783759 26550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:40:09.783944 26545 server_base.cc:1034] running on GCE node
W20260925 22:40:09.783928 26553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:40:09.784494 26545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:40:09.785151 26545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:40:09.786352 26545 hybrid_clock.cc:648] HybridClock initialized: now 1790376009786333 us; error 34 us; skew 500 ppm
Sep 25 22:40:09 dist-test-slave-wjdr krb5kdc[26304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376009, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:40:09.789358 26545 init.cc:377] Logged in from keytab as kudu/127.6.246.194@KRBTEST.COM (short username kudu)
I20260925 22:40:09.790247 26545 webserver.cc:460] Webserver started at http://127.6.246.194:37967/ using document root <none> and password file <none>
I20260925 22:40:09.790782 26545 fs_manager.cc:359] Metadata directory not provided
I20260925 22:40:09.790850 26545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:40:09.791019 26545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:40:09.793200 26545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/minicluster-data/ts-1/data/instance:
uuid: "2dc681d231b54730b3334c5e397dfcd8"
format_stamp: "Formatted at 2026-09-25 22:40:09 on dist-test-slave-wjdr"
server_key: "ded1d4f4fd8f4b207bc644b4ce156c59"
server_key_iv: "8dbdd1ee8f10545b9d5d8b29ef616d01"
server_key_version: "encryptionkey@0"
I20260925 22:40:09.793747 26545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/minicluster-data/ts-1/wal/instance:
uuid: "2dc681d231b54730b3334c5e397dfcd8"
format_stamp: "Formatted at 2026-09-25 22:40:09 on dist-test-slave-wjdr"
server_key: "ded1d4f4fd8f4b207bc644b4ce156c59"
server_key_iv: "8dbdd1ee8f10545b9d5d8b29ef616d01"
server_key_version: "encryptionkey@0"
I20260925 22:40:09.797606 26545 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260925 22:40:09.800015 26560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:09.801276 26545 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 22:40:09.801373 26545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/minicluster-data/ts-1/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/minicluster-data/ts-1/wal
uuid: "2dc681d231b54730b3334c5e397dfcd8"
format_stamp: "Formatted at 2026-09-25 22:40:09 on dist-test-slave-wjdr"
server_key: "ded1d4f4fd8f4b207bc644b4ce156c59"
server_key_iv: "8dbdd1ee8f10545b9d5d8b29ef616d01"
server_key_version: "encryptionkey@0"
I20260925 22:40:09.801510 26545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:40:09.817438 26545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:40:09.821038 26545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:40:09.821256 26545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:40:09.821884 26545 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:40:09.822795 26545 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:40:09.822849 26545 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:09.822916 26545 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:40:09.822947 26545 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:09.832039 26545 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.194:32793
I20260925 22:40:09.832058 26673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.194:32793 every 8 connection(s)
I20260925 22:40:09.832975 26545 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
I20260925 22:40:09.835053 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 26545
I20260925 22:40:09.835182 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/minicluster-data/ts-1/wal/instance
I20260925 22:40:09.835516 7131 external_mini_cluster.cc:1428] Setting key f4fbfeded7a5610a51ec6e9ee43f4673
Sep 25 22:40:09 dist-test-slave-wjdr krb5kdc[26304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376009, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.194@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.246.195@KRBTEST.COM; defaulting to no policy
I20260925 22:40:09.847735 26388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:09.834781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.194:49893 (local address 127.6.246.254:39837)
0925 22:40:09.834945 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:09.834948 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:09.835969 (+ 1021us) server_negotiation.cc:407] Connection header received
0925 22:40:09.837589 (+ 1620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:09.837592 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:09.837644 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:09.837746 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:09.838877 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:09.839396 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:09.840574 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:09.840795 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:09.840919 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:09.843919 (+ 3000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:09.843926 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:09.843930 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:09.843961 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:09.845636 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:09.846167 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:09.846171 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:09.846173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:09.846257 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:09.846665 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:09.846671 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:09.846676 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:09.846891 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:09.846974 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:09.847439 (+ 465us) server_negotiation.cc:299] Negotiation successful
0925 22:40:09.847614 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 22:40:09.848696 26676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:09.835236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:39837 (local address 127.6.246.194:49893)
0925 22:40:09.835776 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:09.835824 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:09.837378 (+ 1554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:09.837916 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:09.837923 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:09.838265 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:40:09.838724 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:09.838735 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:09.839525 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:09.839529 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:09.840386 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:09.840395 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:09.840956 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:09.840960 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:09.841128 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:09.841814 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:40:09.841841 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:40:09.843749 (+ 1908us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:40:09.845774 (+ 2025us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:09.845779 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:09.845793 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:09.846046 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:09.846353 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:09.846355 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:09.846357 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:09.846560 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:09.846986 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:40:09.846991 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:40:09.847216 (+ 225us) client_negotiation.cc:769] Sending connection context
0925 22:40:09.847389 (+ 173us) client_negotiation.cc:240] Negotiation successful
0925 22:40:09.847711 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":143,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.195@KRBTEST.COM" created.
I20260925 22:40:09.850025 26674 heartbeater.cc:347] Connected to a master server at 127.6.246.254:39837
I20260925 22:40:09.850275 26674 heartbeater.cc:464] Registering TS with master...
I20260925 22:40:09.850776 26674 heartbeater.cc:511] Master 127.6.246.254:39837 requested a full tablet report, sending...
I20260925 22:40:09.851872 26349 ts_manager.cc:194] Registered new tserver with Master: 2dc681d231b54730b3334c5e397dfcd8 (127.6.246.194:32793)
I20260925 22:40:09.852533 26349 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.246.194@KRBTEST.COM'} at 127.6.246.194:49893
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.195.keytab.
Entry for principal kudu/127.6.246.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.195.keytab.
I20260925 22:40:09.867533 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.195:0
--local_ip_for_outbound_sockets=127.6.246.195
--webserver_interface=127.6.246.195
--webserver_port=0
--tserver_master_addrs=127.6.246.254:39837
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.195.keytab
--principal=kudu/127.6.246.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:41757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:40:09.978572 26679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:40:09.978823 26679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:40:09.978866 26679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:40:09.982954 26679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:40:09.983011 26679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:40:09.983110 26679 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.195
I20260925 22:40:09.988247 26679 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:41757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.195.keytab
--principal=kudu/127.6.246.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.246.195
--webserver_port=0
--tserver_master_addrs=127.6.246.254:39837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26679
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.195
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:40:09.989286 26679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:40:09.990247 26679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:40:09.999811 26685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:09.999858 26684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:40:10.000174 26679 server_base.cc:1034] running on GCE node
W20260925 22:40:09.999855 26687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:40:10.000734 26679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:40:10.001438 26679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:40:10.002667 26679 hybrid_clock.cc:648] HybridClock initialized: now 1790376010002633 us; error 49 us; skew 500 ppm
Sep 25 22:40:10 dist-test-slave-wjdr krb5kdc[26304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376010, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:40:10.005625 26679 init.cc:377] Logged in from keytab as kudu/127.6.246.195@KRBTEST.COM (short username kudu)
I20260925 22:40:10.006494 26679 webserver.cc:460] Webserver started at http://127.6.246.195:39575/ using document root <none> and password file <none>
I20260925 22:40:10.007025 26679 fs_manager.cc:359] Metadata directory not provided
I20260925 22:40:10.007081 26679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:40:10.007242 26679 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:40:10.009285 26679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/minicluster-data/ts-2/data/instance:
uuid: "c5b899ca0ce84833b2c4fe4c8019bef8"
format_stamp: "Formatted at 2026-09-25 22:40:10 on dist-test-slave-wjdr"
server_key: "b756aea33fa3239bf490af3bda924370"
server_key_iv: "9fc1d19c4eaf667e7899750cd467e524"
server_key_version: "encryptionkey@0"
I20260925 22:40:10.009788 26679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/minicluster-data/ts-2/wal/instance:
uuid: "c5b899ca0ce84833b2c4fe4c8019bef8"
format_stamp: "Formatted at 2026-09-25 22:40:10 on dist-test-slave-wjdr"
server_key: "b756aea33fa3239bf490af3bda924370"
server_key_iv: "9fc1d19c4eaf667e7899750cd467e524"
server_key_version: "encryptionkey@0"
I20260925 22:40:10.013335 26679 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 22:40:10.016182 26694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:10.017608 26679 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 22:40:10.017766 26679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/minicluster-data/ts-2/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/minicluster-data/ts-2/wal
uuid: "c5b899ca0ce84833b2c4fe4c8019bef8"
format_stamp: "Formatted at 2026-09-25 22:40:10 on dist-test-slave-wjdr"
server_key: "b756aea33fa3239bf490af3bda924370"
server_key_iv: "9fc1d19c4eaf667e7899750cd467e524"
server_key_version: "encryptionkey@0"
I20260925 22:40:10.017916 26679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:40:10.030560 26679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:40:10.033504 26679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:40:10.033741 26679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:40:10.034332 26679 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:40:10.035142 26679 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:40:10.035226 26679 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:10.035298 26679 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:40:10.035352 26679 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:10.044051 26679 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.195:33069
I20260925 22:40:10.044164 26807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.195:33069 every 8 connection(s)
I20260925 22:40:10.045542 26679 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
I20260925 22:40:10.054004 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 26679
I20260925 22:40:10.054150 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/minicluster-data/ts-2/wal/instance
I20260925 22:40:10.055048 7131 external_mini_cluster.cc:1428] Setting key 9d7c8489158909b1deba8511f0b8695a
Sep 25 22:40:10 dist-test-slave-wjdr krb5kdc[26304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376010, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.195@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
I20260925 22:40:10.064738 26388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:10.048030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.195:59475 (local address 127.6.246.254:39837)
0925 22:40:10.048191 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:10.048197 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:10.049912 (+ 1715us) server_negotiation.cc:407] Connection header received
0925 22:40:10.051627 (+ 1715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:10.051631 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:10.051691 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:10.051750 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:10.053549 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:10.054480 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:10.056104 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:10.056439 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:10.056561 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:10.060252 (+ 3691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:10.060256 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:10.060259 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:10.060296 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:10.062233 (+ 1937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:10.062931 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:10.062935 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:10.062937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:10.063033 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:10.063424 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:10.063427 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:10.063429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:10.063709 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:10.063816 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:10.064404 (+ 588us) server_negotiation.cc:299] Negotiation successful
0925 22:40:10.064568 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 22:40:10.065744 26810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:10.048741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:39837 (local address 127.6.246.195:59475)
0925 22:40:10.049734 (+ 993us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:10.049798 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:10.051358 (+ 1560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:10.052161 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:10.052169 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:10.052801 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:40:10.053397 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:10.053411 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:10.054643 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:10.054646 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:10.055947 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:10.055959 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:10.056815 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:10.056821 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:10.057054 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:10.058043 (+ 989us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:40:10.058070 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:40:10.060086 (+ 2016us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:40:10.062399 (+ 2313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:10.062409 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:10.062425 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:10.062829 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:10.063139 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:10.063142 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:10.063144 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:10.063304 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:10.063817 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:40:10.063825 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:40:10.064130 (+ 305us) client_negotiation.cc:769] Sending connection context
0925 22:40:10.064347 (+ 217us) client_negotiation.cc:240] Negotiation successful
0925 22:40:10.064705 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":745,"thread_start_us":153,"threads_started":1}
I20260925 22:40:10.067101 26808 heartbeater.cc:347] Connected to a master server at 127.6.246.254:39837
I20260925 22:40:10.067391 26808 heartbeater.cc:464] Registering TS with master...
I20260925 22:40:10.068018 26808 heartbeater.cc:511] Master 127.6.246.254:39837 requested a full tablet report, sending...
I20260925 22:40:10.069296 26349 ts_manager.cc:194] Registered new tserver with Master: c5b899ca0ce84833b2c4fe4c8019bef8 (127.6.246.195:33069)
I20260925 22:40:10.070021 26349 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.246.195@KRBTEST.COM'} at 127.6.246.195:59475
I20260925 22:40:10.071568 7131 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:40:10.081121 26388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:10.073776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59846 (local address 127.6.246.254:39837)
0925 22:40:10.073954 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:10.073958 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:10.074188 (+ 230us) server_negotiation.cc:407] Connection header received
0925 22:40:10.074248 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:10.074250 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:10.074298 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:10.074399 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:10.075031 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:10.075540 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:10.076354 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:10.076655 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:10.076738 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:10.078122 (+ 1384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:10.078125 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:10.078127 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:10.078147 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:10.079457 (+ 1310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:10.079897 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:10.079900 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:10.079901 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:10.079972 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:10.080210 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:10.080212 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:10.080213 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:10.080411 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:10.080592 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:10.080780 (+ 188us) server_negotiation.cc:299] Negotiation successful
0925 22:40:10.080938 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 22:40:10.084069 26349 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59846:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:40:10.087239 26349 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:40:10.101555 26819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:10.095492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.194:32793 (local address 127.0.0.1:53308)
0925 22:40:10.095886 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:10.095937 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:10.096318 (+ 381us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:10.097158 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:10.097164 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:10.097188 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:40:10.097331 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:10.097337 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:10.098617 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:10.098621 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:10.099977 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:10.099988 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:10.101034 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:10.101037 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:10.101138 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:10.101172 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 22:40:10.101287 (+ 115us) client_negotiation.cc:240] Negotiation successful
0925 22:40:10.101408 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"mutex_wait_us":67,"thread_start_us":89,"threads_started":1}
I20260925 22:40:10.101567 26820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:10.095647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53308 (local address 127.6.246.194:32793)
0925 22:40:10.096130 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:10.096142 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:10.096169 (+ 27us) server_negotiation.cc:407] Connection header received
0925 22:40:10.096576 (+ 407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:10.096582 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:10.096834 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:10.096991 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:40:10.097462 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:10.098471 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:10.100110 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:10.100903 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:10.101035 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:10.101188 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:10.101324 (+ 136us) server_negotiation.cc:299] Negotiation successful
0925 22:40:10.101416 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":118,"threads_started":1}
I20260925 22:40:10.101933 26821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:10.095935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.195:33069 (local address 127.0.0.1:36760)
0925 22:40:10.096346 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:10.096358 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:10.096440 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:10.097885 (+ 1445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:10.097887 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:10.097905 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:40:10.097977 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:10.097982 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:10.099280 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:10.099284 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:10.100502 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:10.100512 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:10.101550 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:10.101553 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:10.101647 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:10.101670 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 22:40:10.101732 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 22:40:10.101797 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"mutex_wait_us":16,"thread_start_us":87,"threads_started":1}
I20260925 22:40:10.102100 26824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:10.096796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36760 (local address 127.6.246.195:33069)
0925 22:40:10.097370 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:10.097384 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:10.097405 (+ 21us) server_negotiation.cc:407] Connection header received
0925 22:40:10.097527 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:10.097531 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:10.097735 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:10.097887 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:40:10.098098 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:10.099146 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:10.100639 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:10.101424 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:10.101662 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:10.101815 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:10.101862 (+ 47us) server_negotiation.cc:299] Negotiation successful
0925 22:40:10.101946 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":152,"threads_started":1}
I20260925 22:40:10.103845 26822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:10.096173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.193:38643 (local address 127.0.0.1:45612)
0925 22:40:10.096505 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:10.096519 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:10.096595 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:10.097493 (+ 898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:10.097496 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:10.097517 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:40:10.097591 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:10.097597 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:10.099010 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:10.099013 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:10.100238 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:10.100245 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:10.101530 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:10.101534 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:10.103524 (+ 1990us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:10.103546 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 22:40:10.103647 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 22:40:10.103720 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"mutex_wait_us":7,"thread_start_us":88,"threads_started":1}
I20260925 22:40:10.103924 26823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:10.096728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45612 (local address 127.6.246.193:38643)
0925 22:40:10.097033 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:10.097049 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:10.097070 (+ 21us) server_negotiation.cc:407] Connection header received
0925 22:40:10.097143 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:10.097148 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:10.097320 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:10.097466 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:40:10.097712 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:10.098859 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:10.100362 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:10.101205 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:10.101329 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:10.101463 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:10.103680 (+ 2217us) server_negotiation.cc:299] Negotiation successful
0925 22:40:10.103773 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":138,"threads_started":1}
I20260925 22:40:10.105289 26608 tablet_service.cc:1467] Processing CreateTablet for tablet 2f1ea1c350754c72ad8707a9fe199411 (DEFAULT_TABLE table=test-table [id=bb9bed11daf1420189e8aba54cc3dea7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:40:10.105286 26742 tablet_service.cc:1467] Processing CreateTablet for tablet 2f1ea1c350754c72ad8707a9fe199411 (DEFAULT_TABLE table=test-table [id=bb9bed11daf1420189e8aba54cc3dea7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:40:10.106413 26742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f1ea1c350754c72ad8707a9fe199411. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:40:10.106410 26608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f1ea1c350754c72ad8707a9fe199411. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:40:10.111025 26474 tablet_service.cc:1467] Processing CreateTablet for tablet 2f1ea1c350754c72ad8707a9fe199411 (DEFAULT_TABLE table=test-table [id=bb9bed11daf1420189e8aba54cc3dea7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:40:10.111945 26474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f1ea1c350754c72ad8707a9fe199411. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:40:10.114001 26826 tablet_bootstrap.cc:492] T 2f1ea1c350754c72ad8707a9fe199411 P 2dc681d231b54730b3334c5e397dfcd8: Bootstrap starting.
I20260925 22:40:10.114434 26825 tablet_bootstrap.cc:492] T 2f1ea1c350754c72ad8707a9fe199411 P c5b899ca0ce84833b2c4fe4c8019bef8: Bootstrap starting.
I20260925 22:40:10.116812 26825 tablet_bootstrap.cc:654] T 2f1ea1c350754c72ad8707a9fe199411 P c5b899ca0ce84833b2c4fe4c8019bef8: Neither blocks nor log segments found. Creating new log.
I20260925 22:40:10.118100 26825 log.cc:824] T 2f1ea1c350754c72ad8707a9fe199411 P c5b899ca0ce84833b2c4fe4c8019bef8: Log is configured to *not* fsync() on all Append() calls
I20260925 22:40:10.118852 26826 tablet_bootstrap.cc:654] T 2f1ea1c350754c72ad8707a9fe199411 P 2dc681d231b54730b3334c5e397dfcd8: Neither blocks nor log segments found. Creating new log.
I20260925 22:40:10.119583 26826 log.cc:824] T 2f1ea1c350754c72ad8707a9fe199411 P 2dc681d231b54730b3334c5e397dfcd8: Log is configured to *not* fsync() on all Append() calls
I20260925 22:40:10.120386 26825 tablet_bootstrap.cc:492] T 2f1ea1c350754c72ad8707a9fe199411 P c5b899ca0ce84833b2c4fe4c8019bef8: No bootstrap required, opened a new log
I20260925 22:40:10.120613 26825 ts_tablet_manager.cc:1397] T 2f1ea1c350754c72ad8707a9fe199411 P c5b899ca0ce84833b2c4fe4c8019bef8: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260925 22:40:10.121446 26826 tablet_bootstrap.cc:492] T 2f1ea1c350754c72ad8707a9fe199411 P 2dc681d231b54730b3334c5e397dfcd8: No bootstrap required, opened a new log
I20260925 22:40:10.121644 26826 ts_tablet_manager.cc:1397] T 2f1ea1c350754c72ad8707a9fe199411 P 2dc681d231b54730b3334c5e397dfcd8: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260925 22:40:10.123883 26825 raft_consensus.cc:348] T 2f1ea1c350754c72ad8707a9fe199411 P c5b899ca0ce84833b2c4fe4c8019bef8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dc681d231b54730b3334c5e397dfcd8" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 32793 } } peers { permanent_uuid: "5843a00aa3d74373a1ba83c86f80e02d" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 38643 } } peers { permanent_uuid: "c5b899ca0ce84833b2c4fe4c8019bef8" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 33069 } }
I20260925 22:40:10.124218 26825 raft_consensus.cc:374] T 2f1ea1c350754c72ad8707a9fe199411 P c5b899ca0ce84833b2c4fe4c8019bef8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:40:10.124289 26825 raft_consensus.cc:729] T 2f1ea1c350754c72ad8707a9fe199411 P c5b899ca0ce84833b2c4fe4c8019bef8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5b899ca0ce84833b2c4fe4c8019bef8, State: Initialized, Role: FOLLOWER
I20260925 22:40:10.124702 26828 tablet_bootstrap.cc:492] T 2f1ea1c350754c72ad8707a9fe199411 P 5843a00aa3d74373a1ba83c86f80e02d: Bootstrap starting.
I20260925 22:40:10.124780 26825 consensus_queue.cc:260] T 2f1ea1c350754c72ad8707a9fe199411 P c5b899ca0ce84833b2c4fe4c8019bef8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dc681d231b54730b3334c5e397dfcd8" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 32793 } } peers { permanent_uuid: "5843a00aa3d74373a1ba83c86f80e02d" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 38643 } } peers { permanent_uuid: "c5b899ca0ce84833b2c4fe4c8019bef8" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 33069 } }
I20260925 22:40:10.125304 26826 raft_consensus.cc:348] T 2f1ea1c350754c72ad8707a9fe199411 P 2dc681d231b54730b3334c5e397dfcd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dc681d231b54730b3334c5e397dfcd8" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 32793 } } peers { permanent_uuid: "5843a00aa3d74373a1ba83c86f80e02d" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 38643 } } peers { permanent_uuid: "c5b899ca0ce84833b2c4fe4c8019bef8" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 33069 } }
I20260925 22:40:10.125528 26826 raft_consensus.cc:374] T 2f1ea1c350754c72ad8707a9fe199411 P 2dc681d231b54730b3334c5e397dfcd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:40:10.125586 26826 raft_consensus.cc:729] T 2f1ea1c350754c72ad8707a9fe199411 P 2dc681d231b54730b3334c5e397dfcd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dc681d231b54730b3334c5e397dfcd8, State: Initialized, Role: FOLLOWER
I20260925 22:40:10.125773 26825 ts_tablet_manager.cc:1428] T 2f1ea1c350754c72ad8707a9fe199411 P c5b899ca0ce84833b2c4fe4c8019bef8: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260925 22:40:10.126024 26826 consensus_queue.cc:260] T 2f1ea1c350754c72ad8707a9fe199411 P 2dc681d231b54730b3334c5e397dfcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dc681d231b54730b3334c5e397dfcd8" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 32793 } } peers { permanent_uuid: "5843a00aa3d74373a1ba83c86f80e02d" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 38643 } } peers { permanent_uuid: "c5b899ca0ce84833b2c4fe4c8019bef8" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 33069 } }
I20260925 22:40:10.126142 26808 heartbeater.cc:503] Master 127.6.246.254:39837 was elected leader, sending a full tablet report...
I20260925 22:40:10.126832 26674 heartbeater.cc:503] Master 127.6.246.254:39837 was elected leader, sending a full tablet report...
I20260925 22:40:10.126892 26826 ts_tablet_manager.cc:1428] T 2f1ea1c350754c72ad8707a9fe199411 P 2dc681d231b54730b3334c5e397dfcd8: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260925 22:40:10.126955 26828 tablet_bootstrap.cc:654] T 2f1ea1c350754c72ad8707a9fe199411 P 5843a00aa3d74373a1ba83c86f80e02d: Neither blocks nor log segments found. Creating new log.
I20260925 22:40:10.127767 26828 log.cc:824] T 2f1ea1c350754c72ad8707a9fe199411 P 5843a00aa3d74373a1ba83c86f80e02d: Log is configured to *not* fsync() on all Append() calls
I20260925 22:40:10.129889 26828 tablet_bootstrap.cc:492] T 2f1ea1c350754c72ad8707a9fe199411 P 5843a00aa3d74373a1ba83c86f80e02d: No bootstrap required, opened a new log
I20260925 22:40:10.130084 26828 ts_tablet_manager.cc:1397] T 2f1ea1c350754c72ad8707a9fe199411 P 5843a00aa3d74373a1ba83c86f80e02d: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20260925 22:40:10.133615 26828 raft_consensus.cc:348] T 2f1ea1c350754c72ad8707a9fe199411 P 5843a00aa3d74373a1ba83c86f80e02d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dc681d231b54730b3334c5e397dfcd8" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 32793 } } peers { permanent_uuid: "5843a00aa3d74373a1ba83c86f80e02d" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 38643 } } peers { permanent_uuid: "c5b899ca0ce84833b2c4fe4c8019bef8" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 33069 } }
I20260925 22:40:10.133899 26828 raft_consensus.cc:374] T 2f1ea1c350754c72ad8707a9fe199411 P 5843a00aa3d74373a1ba83c86f80e02d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:40:10.133960 26828 raft_consensus.cc:729] T 2f1ea1c350754c72ad8707a9fe199411 P 5843a00aa3d74373a1ba83c86f80e02d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5843a00aa3d74373a1ba83c86f80e02d, State: Initialized, Role: FOLLOWER
I20260925 22:40:10.134352 26828 consensus_queue.cc:260] T 2f1ea1c350754c72ad8707a9fe199411 P 5843a00aa3d74373a1ba83c86f80e02d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dc681d231b54730b3334c5e397dfcd8" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 32793 } } peers { permanent_uuid: "5843a00aa3d74373a1ba83c86f80e02d" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 38643 } } peers { permanent_uuid: "c5b899ca0ce84833b2c4fe4c8019bef8" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 33069 } }
I20260925 22:40:10.135002 26540 heartbeater.cc:503] Master 127.6.246.254:39837 was elected leader, sending a full tablet report...
I20260925 22:40:10.135318 26828 ts_tablet_manager.cc:1428] T 2f1ea1c350754c72ad8707a9fe199411 P 5843a00aa3d74373a1ba83c86f80e02d: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260925 22:40:10.299218 26809 tablet.cc:2369] T 2f1ea1c350754c72ad8707a9fe199411 P c5b899ca0ce84833b2c4fe4c8019bef8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 22:40:10.334188 26675 tablet.cc:2369] T 2f1ea1c350754c72ad8707a9fe199411 P 2dc681d231b54730b3334c5e397dfcd8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 22:40:10.359287 26541 tablet.cc:2369] T 2f1ea1c350754c72ad8707a9fe199411 P 5843a00aa3d74373a1ba83c86f80e02d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:40:10.396858 26831 raft_consensus.cc:482] T 2f1ea1c350754c72ad8707a9fe199411 P 2dc681d231b54730b3334c5e397dfcd8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:40:10.397120 26831 raft_consensus.cc:504] T 2f1ea1c350754c72ad8707a9fe199411 P 2dc681d231b54730b3334c5e397dfcd8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dc681d231b54730b3334c5e397dfcd8" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 32793 } } peers { permanent_uuid: "5843a00aa3d74373a1ba83c86f80e02d" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 38643 } } peers { permanent_uuid: "c5b899ca0ce84833b2c4fe4c8019bef8" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 33069 } }
I20260925 22:40:10.398265 26831 leader_election.cc:288] T 2f1ea1c350754c72ad8707a9fe199411 P 2dc681d231b54730b3334c5e397dfcd8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5843a00aa3d74373a1ba83c86f80e02d (127.6.246.193:38643), c5b899ca0ce84833b2c4fe4c8019bef8 (127.6.246.195:33069)
I20260925 22:40:10.403622 26823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:10.398707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.194:53661 (local address 127.6.246.193:38643)
0925 22:40:10.398855 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:10.398859 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:10.399102 (+ 243us) server_negotiation.cc:407] Connection header received
0925 22:40:10.399211 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:10.399214 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:10.399274 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:10.399367 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:40:10.399763 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:10.400592 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:10.402344 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:10.402952 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:10.403076 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:10.403125 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:10.403430 (+ 305us) server_negotiation.cc:299] Negotiation successful
0925 22:40:10.403495 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 22:40:10.403622 26824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:10.398696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.194:55613 (local address 127.6.246.195:33069)
0925 22:40:10.398834 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:10.398841 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:10.399101 (+ 260us) server_negotiation.cc:407] Connection header received
0925 22:40:10.399211 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:10.399214 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:10.399268 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:10.399365 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:40:10.399764 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:10.400592 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:10.402344 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:10.402952 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:10.403075 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:10.403125 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:10.403430 (+ 305us) server_negotiation.cc:299] Negotiation successful
0925 22:40:10.403495 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 22:40:10.403702 26835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:10.398552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.193:38643 (local address 127.6.246.194:53661)
0925 22:40:10.398971 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:10.398983 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:10.399114 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:10.399399 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:10.399402 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:10.399429 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:40:10.399612 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:10.399620 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:10.400720 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:10.400724 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:10.402208 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:10.402220 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:10.403086 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:10.403090 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:10.403259 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:10.403298 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 22:40:10.403397 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 22:40:10.403508 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"mutex_wait_us":254,"spinlock_wait_cycles":25600,"thread_start_us":91,"threads_started":1}
I20260925 22:40:10.403707 26834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:10.398552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.195:33069 (local address 127.6.246.194:55613)
0925 22:40:10.398958 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:10.398973 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:10.399113 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:10.399399 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:10.399402 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:10.399429 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:40:10.399616 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:10.399621 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:10.400720 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:10.400724 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:10.402208 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:10.402220 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:10.403086 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:10.403090 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:10.403254 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:10.403315 (+ 61us) client_negotiation.cc:769] Sending connection context
0925 22:40:10.403397 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 22:40:10.403508 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"mutex_wait_us":206,"spinlock_wait_cycles":18048,"thread_start_us":92,"threads_started":1}
I20260925 22:40:10.404377 26494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f1ea1c350754c72ad8707a9fe199411" candidate_uuid: "2dc681d231b54730b3334c5e397dfcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5843a00aa3d74373a1ba83c86f80e02d" is_pre_election: true
I20260925 22:40:10.404376 26762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f1ea1c350754c72ad8707a9fe199411" candidate_uuid: "2dc681d231b54730b3334c5e397dfcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5b899ca0ce84833b2c4fe4c8019bef8" is_pre_election: true
I20260925 22:40:10.404718 26494 raft_consensus.cc:2454] T 2f1ea1c350754c72ad8707a9fe199411 P 5843a00aa3d74373a1ba83c86f80e02d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2dc681d231b54730b3334c5e397dfcd8 in term 0.
I20260925 22:40:10.404731 26762 raft_consensus.cc:2454] T 2f1ea1c350754c72ad8707a9fe199411 P c5b899ca0ce84833b2c4fe4c8019bef8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2dc681d231b54730b3334c5e397dfcd8 in term 0.
I20260925 22:40:10.405232 26564 leader_election.cc:302] T 2f1ea1c350754c72ad8707a9fe199411 P 2dc681d231b54730b3334c5e397dfcd8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2dc681d231b54730b3334c5e397dfcd8, 5843a00aa3d74373a1ba83c86f80e02d; no voters:
I20260925 22:40:10.405827 26831 raft_consensus.cc:2781] T 2f1ea1c350754c72ad8707a9fe199411 P 2dc681d231b54730b3334c5e397dfcd8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:40:10.405932 26831 raft_consensus.cc:482] T 2f1ea1c350754c72ad8707a9fe199411 P 2dc681d231b54730b3334c5e397dfcd8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:40:10.405999 26831 raft_consensus.cc:3037] T 2f1ea1c350754c72ad8707a9fe199411 P 2dc681d231b54730b3334c5e397dfcd8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:40:10.407156 26831 raft_consensus.cc:504] T 2f1ea1c350754c72ad8707a9fe199411 P 2dc681d231b54730b3334c5e397dfcd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dc681d231b54730b3334c5e397dfcd8" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 32793 } } peers { permanent_uuid: "5843a00aa3d74373a1ba83c86f80e02d" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 38643 } } peers { permanent_uuid: "c5b899ca0ce84833b2c4fe4c8019bef8" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 33069 } }
I20260925 22:40:10.407452 26831 leader_election.cc:288] T 2f1ea1c350754c72ad8707a9fe199411 P 2dc681d231b54730b3334c5e397dfcd8 [CANDIDATE]: Term 1 election: Requested vote from peers 5843a00aa3d74373a1ba83c86f80e02d (127.6.246.193:38643), c5b899ca0ce84833b2c4fe4c8019bef8 (127.6.246.195:33069)
I20260925 22:40:10.407845 26762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f1ea1c350754c72ad8707a9fe199411" candidate_uuid: "2dc681d231b54730b3334c5e397dfcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5b899ca0ce84833b2c4fe4c8019bef8"
I20260925 22:40:10.407845 26494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f1ea1c350754c72ad8707a9fe199411" candidate_uuid: "2dc681d231b54730b3334c5e397dfcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5843a00aa3d74373a1ba83c86f80e02d"
I20260925 22:40:10.407979 26494 raft_consensus.cc:3037] T 2f1ea1c350754c72ad8707a9fe199411 P 5843a00aa3d74373a1ba83c86f80e02d [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:40:10.408005 26762 raft_consensus.cc:3037] T 2f1ea1c350754c72ad8707a9fe199411 P c5b899ca0ce84833b2c4fe4c8019bef8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:40:10.409294 26494 raft_consensus.cc:2454] T 2f1ea1c350754c72ad8707a9fe199411 P 5843a00aa3d74373a1ba83c86f80e02d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2dc681d231b54730b3334c5e397dfcd8 in term 1.
I20260925 22:40:10.409293 26762 raft_consensus.cc:2454] T 2f1ea1c350754c72ad8707a9fe199411 P c5b899ca0ce84833b2c4fe4c8019bef8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2dc681d231b54730b3334c5e397dfcd8 in term 1.
I20260925 22:40:10.409607 26564 leader_election.cc:302] T 2f1ea1c350754c72ad8707a9fe199411 P 2dc681d231b54730b3334c5e397dfcd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2dc681d231b54730b3334c5e397dfcd8, 5843a00aa3d74373a1ba83c86f80e02d; no voters:
I20260925 22:40:10.409838 26831 raft_consensus.cc:2781] T 2f1ea1c350754c72ad8707a9fe199411 P 2dc681d231b54730b3334c5e397dfcd8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:40:10.410076 26831 raft_consensus.cc:686] T 2f1ea1c350754c72ad8707a9fe199411 P 2dc681d231b54730b3334c5e397dfcd8 [term 1 LEADER]: Becoming Leader. State: Replica: 2dc681d231b54730b3334c5e397dfcd8, State: Running, Role: LEADER
I20260925 22:40:10.410414 26831 consensus_queue.cc:237] T 2f1ea1c350754c72ad8707a9fe199411 P 2dc681d231b54730b3334c5e397dfcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dc681d231b54730b3334c5e397dfcd8" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 32793 } } peers { permanent_uuid: "5843a00aa3d74373a1ba83c86f80e02d" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 38643 } } peers { permanent_uuid: "c5b899ca0ce84833b2c4fe4c8019bef8" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 33069 } }
I20260925 22:40:10.413442 26348 catalog_manager.cc:5495] T 2f1ea1c350754c72ad8707a9fe199411 P 2dc681d231b54730b3334c5e397dfcd8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2dc681d231b54730b3334c5e397dfcd8 (127.6.246.194). New cstate: current_term: 1 leader_uuid: "2dc681d231b54730b3334c5e397dfcd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dc681d231b54730b3334c5e397dfcd8" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 32793 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5843a00aa3d74373a1ba83c86f80e02d" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 38643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5b899ca0ce84833b2c4fe4c8019bef8" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 33069 } health_report { overall_health: UNKNOWN } } }
I20260925 22:40:10.472347 26820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:10.468101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53324 (local address 127.6.246.194:32793)
0925 22:40:10.468283 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:10.468288 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:10.468303 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:40:10.468404 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:10.468406 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:10.468582 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:10.468691 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:40:10.468976 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:10.469542 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:10.470461 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:10.470692 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:10.470780 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:10.471165 (+ 385us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:40:10.471802 (+ 637us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:40:10.471918 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:10.471993 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 22:40:10.472163 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 22:40:10.480028 26762 raft_consensus.cc:1261] T 2f1ea1c350754c72ad8707a9fe199411 P c5b899ca0ce84833b2c4fe4c8019bef8 [term 1 FOLLOWER]: Refusing update from remote peer 2dc681d231b54730b3334c5e397dfcd8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:40:10.480028 26494 raft_consensus.cc:1261] T 2f1ea1c350754c72ad8707a9fe199411 P 5843a00aa3d74373a1ba83c86f80e02d [term 1 FOLLOWER]: Refusing update from remote peer 2dc681d231b54730b3334c5e397dfcd8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:40:10.480909 26836 consensus_queue.cc:1035] T 2f1ea1c350754c72ad8707a9fe199411 P 2dc681d231b54730b3334c5e397dfcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5843a00aa3d74373a1ba83c86f80e02d" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 38643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:40:10.481107 26831 consensus_queue.cc:1035] T 2f1ea1c350754c72ad8707a9fe199411 P 2dc681d231b54730b3334c5e397dfcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5b899ca0ce84833b2c4fe4c8019bef8" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 33069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:40:10.489894 26840 mvcc.cc:205] Tried to move back new op lower bound from 7333380138916618240 to 7333380138643603456. Current Snapshot: MvccSnapshot[applied={T|T < 7333380138916618240}]
I20260925 22:40:10.492771 26841 mvcc.cc:205] Tried to move back new op lower bound from 7333380138916618240 to 7333380138643603456. Current Snapshot: MvccSnapshot[applied={T|T < 7333380138916618240}]
I20260925 22:40:10.499676 26348 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59846:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:40:10.499927 26348 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59846:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:40:10.502758 26348 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c6dbbadfc1be459895dff1ea035c74d5: Sending DeleteTablet for 3 replicas of tablet 2f1ea1c350754c72ad8707a9fe199411
I20260925 22:40:10.503839 26742 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f1ea1c350754c72ad8707a9fe199411 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:40:10 UTC) from {username='kudu', principal='kudu/127.6.246.254@KRBTEST.COM'} at 127.0.0.1:36760
I20260925 22:40:10.503822 26608 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f1ea1c350754c72ad8707a9fe199411 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:40:10 UTC) from {username='kudu', principal='kudu/127.6.246.254@KRBTEST.COM'} at 127.0.0.1:53308
I20260925 22:40:10.504521 26850 tablet_replica.cc:330] T 2f1ea1c350754c72ad8707a9fe199411 P 2dc681d231b54730b3334c5e397dfcd8: stopping tablet replica
I20260925 22:40:10.504513 26849 tablet_replica.cc:330] T 2f1ea1c350754c72ad8707a9fe199411 P c5b899ca0ce84833b2c4fe4c8019bef8: stopping tablet replica
I20260925 22:40:10.504565 26474 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f1ea1c350754c72ad8707a9fe199411 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:40:10 UTC) from {username='kudu', principal='kudu/127.6.246.254@KRBTEST.COM'} at 127.0.0.1:45612
I20260925 22:40:10.504971 26850 raft_consensus.cc:2229] T 2f1ea1c350754c72ad8707a9fe199411 P 2dc681d231b54730b3334c5e397dfcd8 [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:40:10.505012 26849 raft_consensus.cc:2229] T 2f1ea1c350754c72ad8707a9fe199411 P c5b899ca0ce84833b2c4fe4c8019bef8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:40:10.505421 26850 raft_consensus.cc:2258] T 2f1ea1c350754c72ad8707a9fe199411 P 2dc681d231b54730b3334c5e397dfcd8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:40:10.505435 26851 tablet_replica.cc:330] T 2f1ea1c350754c72ad8707a9fe199411 P 5843a00aa3d74373a1ba83c86f80e02d: stopping tablet replica
I20260925 22:40:10.505673 26849 raft_consensus.cc:2258] T 2f1ea1c350754c72ad8707a9fe199411 P c5b899ca0ce84833b2c4fe4c8019bef8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:40:10.505743 26851 raft_consensus.cc:2229] T 2f1ea1c350754c72ad8707a9fe199411 P 5843a00aa3d74373a1ba83c86f80e02d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:40:10.505923 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 26411
I20260925 22:40:10.507400 26850 ts_tablet_manager.cc:1905] T 2f1ea1c350754c72ad8707a9fe199411 P 2dc681d231b54730b3334c5e397dfcd8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:40:10.509433 26849 ts_tablet_manager.cc:1905] T 2f1ea1c350754c72ad8707a9fe199411 P c5b899ca0ce84833b2c4fe4c8019bef8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:40:10.510617 26850 ts_tablet_manager.cc:1918] T 2f1ea1c350754c72ad8707a9fe199411 P 2dc681d231b54730b3334c5e397dfcd8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:40:10.510727 26850 log.cc:1196] T 2f1ea1c350754c72ad8707a9fe199411 P 2dc681d231b54730b3334c5e397dfcd8: Deleting WAL directory at /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/minicluster-data/ts-1/wal/wals/2f1ea1c350754c72ad8707a9fe199411
I20260925 22:40:10.511078 26850 ts_tablet_manager.cc:1939] T 2f1ea1c350754c72ad8707a9fe199411 P 2dc681d231b54730b3334c5e397dfcd8: Deleting consensus metadata
I20260925 22:40:10.512740 26336 catalog_manager.cc:4841] TS 2dc681d231b54730b3334c5e397dfcd8 (127.6.246.194:32793): tablet 2f1ea1c350754c72ad8707a9fe199411 (table test-table [id=bb9bed11daf1420189e8aba54cc3dea7]) successfully deleted
I20260925 22:40:10.512830 26849 ts_tablet_manager.cc:1918] T 2f1ea1c350754c72ad8707a9fe199411 P c5b899ca0ce84833b2c4fe4c8019bef8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:40:10.513063 26849 log.cc:1196] T 2f1ea1c350754c72ad8707a9fe199411 P c5b899ca0ce84833b2c4fe4c8019bef8: Deleting WAL directory at /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790375752797504-7131-0/minicluster-data/ts-2/wal/wals/2f1ea1c350754c72ad8707a9fe199411
I20260925 22:40:10.513407 26849 ts_tablet_manager.cc:1939] T 2f1ea1c350754c72ad8707a9fe199411 P c5b899ca0ce84833b2c4fe4c8019bef8: Deleting consensus metadata
I20260925 22:40:10.514816 26333 catalog_manager.cc:4841] TS c5b899ca0ce84833b2c4fe4c8019bef8 (127.6.246.195:33069): tablet 2f1ea1c350754c72ad8707a9fe199411 (table test-table [id=bb9bed11daf1420189e8aba54cc3dea7]) successfully deleted
W20260925 22:40:10.516911 26336 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.246.193:38643 (error 108)
I20260925 22:40:10.517958 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 26545
I20260925 22:40:10.518085 26822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:10.517754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.193:38643 (local address 127.0.0.1:45614)
0925 22:40:10.517905 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:10.517982 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.193:38643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260925 22:40:10.520130 26336 catalog_manager.cc:4584] TS 5843a00aa3d74373a1ba83c86f80e02d (127.6.246.193:38643): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2f1ea1c350754c72ad8707a9fe199411: Network error: Client connection negotiation failed: client connection to 127.6.246.193:38643: connect: Connection refused (error 111)
I20260925 22:40:10.525120 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 26679
I20260925 22:40:10.533412 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 26317
2026-09-25T22:40:10Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2587 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-25T22:40:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:40:10Z Disabled control of system clock
I20260925 22:40:10.568015 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790375752797504-7131-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790375752797504-7131-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790375752797504-7131-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:45707
--webserver_interface=127.6.246.254
--webserver_port=0
--builtin_ntp_servers=127.6.246.212:45947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.246.254:45707
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 22:40:10.696220 26856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:40:10.696614 26856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:40:10.696657 26856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:40:10.702224 26856 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:40:10.702330 26856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:40:10.702363 26856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:40:10.702396 26856 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:40:10.702432 26856 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:40:10.708352 26856 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:45947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790375752797504-7131-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790375752797504-7131-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.246.254:45707
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:45707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.246.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790375752797504-7131-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:40:10.709780 26856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:40:10.711499 26856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:40:10.721518 26861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:10.721491 26864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:10.721521 26862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:40:10.721660 26856 server_base.cc:1034] running on GCE node
I20260925 22:40:10.722527 26856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:40:10.723878 26856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:40:10.725095 26856 hybrid_clock.cc:648] HybridClock initialized: now 1790376010725069 us; error 47 us; skew 500 ppm
I20260925 22:40:10.727160 26856 webserver.cc:460] Webserver started at http://127.6.246.254:37599/ using document root <none> and password file <none>
I20260925 22:40:10.727842 26856 fs_manager.cc:359] Metadata directory not provided
I20260925 22:40:10.727941 26856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:40:10.728212 26856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:40:10.730631 26856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790375752797504-7131-0/minicluster-data/master-0/data/instance:
uuid: "8d8b83e1b70d41548334b38ed94234cd"
format_stamp: "Formatted at 2026-09-25 22:40:10 on dist-test-slave-wjdr"
server_key: "072f2d5409ef2ad51fb4de81828cf357"
server_key_iv: "7c591e1caa429dc0d6d1181e1f8416b5"
server_key_version: "encryptionkey@0"
I20260925 22:40:10.731262 26856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790375752797504-7131-0/minicluster-data/master-0/wal/instance:
uuid: "8d8b83e1b70d41548334b38ed94234cd"
format_stamp: "Formatted at 2026-09-25 22:40:10 on dist-test-slave-wjdr"
server_key: "072f2d5409ef2ad51fb4de81828cf357"
server_key_iv: "7c591e1caa429dc0d6d1181e1f8416b5"
server_key_version: "encryptionkey@0"
I20260925 22:40:10.735466 26856 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 22:40:10.738274 26870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:10.739631 26856 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:40:10.739778 26856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790375752797504-7131-0/minicluster-data/master-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790375752797504-7131-0/minicluster-data/master-0/wal
uuid: "8d8b83e1b70d41548334b38ed94234cd"
format_stamp: "Formatted at 2026-09-25 22:40:10 on dist-test-slave-wjdr"
server_key: "072f2d5409ef2ad51fb4de81828cf357"
server_key_iv: "7c591e1caa429dc0d6d1181e1f8416b5"
server_key_version: "encryptionkey@0"
I20260925 22:40:10.739902 26856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790375752797504-7131-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790375752797504-7131-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790375752797504-7131-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:40:10.754014 26856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:40:10.755071 26856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:40:10.755324 26856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:40:10.763852 26856 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.254:45707
I20260925 22:40:10.763907 26922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.254:45707 every 8 connection(s)
I20260925 22:40:10.765193 26856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
I20260925 22:40:10.769076 26923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:40:10.775350 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 26856
I20260925 22:40:10.775537 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790375752797504-7131-0/minicluster-data/master-0/wal/instance
I20260925 22:40:10.775875 7131 external_mini_cluster.cc:1428] Setting key 2d05077e23c500ff359ef4aba8a6d97d
I20260925 22:40:10.776211 26923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d8b83e1b70d41548334b38ed94234cd: Bootstrap starting.
I20260925 22:40:10.779541 26923 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d8b83e1b70d41548334b38ed94234cd: Neither blocks nor log segments found. Creating new log.
I20260925 22:40:10.780475 26923 log.cc:824] T 00000000000000000000000000000000 P 8d8b83e1b70d41548334b38ed94234cd: Log is configured to *not* fsync() on all Append() calls
I20260925 22:40:10.782729 26923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d8b83e1b70d41548334b38ed94234cd: No bootstrap required, opened a new log
I20260925 22:40:10.786684 26926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:10.777632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51376 (local address 127.6.246.254:45707)
0925 22:40:10.778491 (+ 859us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:10.778517 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:10.778559 (+ 42us) server_negotiation.cc:407] Connection header received
0925 22:40:10.779351 (+ 792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:10.779370 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:10.779922 (+ 552us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:10.780313 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:10.780859 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:10.781901 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:10.782859 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:10.783376 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:10.783570 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:10.784680 (+ 1110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:10.784685 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:10.784699 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:40:10.785145 (+ 446us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:10.785210 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:40:10.785231 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:10.785358 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:10.785509 (+ 151us) server_negotiation.cc:299] Negotiation successful
0925 22:40:10.785821 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":231,"threads_started":1}
I20260925 22:40:10.787161 26923 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d8b83e1b70d41548334b38ed94234cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d8b83e1b70d41548334b38ed94234cd" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 45707 } }
I20260925 22:40:10.787459 26923 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d8b83e1b70d41548334b38ed94234cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:40:10.787519 26923 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d8b83e1b70d41548334b38ed94234cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d8b83e1b70d41548334b38ed94234cd, State: Initialized, Role: FOLLOWER
I20260925 22:40:10.788043 26923 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d8b83e1b70d41548334b38ed94234cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d8b83e1b70d41548334b38ed94234cd" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 45707 } }
I20260925 22:40:10.788187 26923 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d8b83e1b70d41548334b38ed94234cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:40:10.788264 26923 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d8b83e1b70d41548334b38ed94234cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:40:10.788393 26923 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d8b83e1b70d41548334b38ed94234cd [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:40:10.789700 26923 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d8b83e1b70d41548334b38ed94234cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d8b83e1b70d41548334b38ed94234cd" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 45707 } }
I20260925 22:40:10.790100 26923 leader_election.cc:302] T 00000000000000000000000000000000 P 8d8b83e1b70d41548334b38ed94234cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d8b83e1b70d41548334b38ed94234cd; no voters:
I20260925 22:40:10.790441 26923 leader_election.cc:288] T 00000000000000000000000000000000 P 8d8b83e1b70d41548334b38ed94234cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:40:10.790597 26928 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d8b83e1b70d41548334b38ed94234cd [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:40:10.790866 26928 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d8b83e1b70d41548334b38ed94234cd [term 1 LEADER]: Becoming Leader. State: Replica: 8d8b83e1b70d41548334b38ed94234cd, State: Running, Role: LEADER
I20260925 22:40:10.791217 26928 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d8b83e1b70d41548334b38ed94234cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d8b83e1b70d41548334b38ed94234cd" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 45707 } }
I20260925 22:40:10.791914 26923 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d8b83e1b70d41548334b38ed94234cd [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:40:10.793186 26930 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d8b83e1b70d41548334b38ed94234cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d8b83e1b70d41548334b38ed94234cd. Latest consensus state: current_term: 1 leader_uuid: "8d8b83e1b70d41548334b38ed94234cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d8b83e1b70d41548334b38ed94234cd" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 45707 } } }
I20260925 22:40:10.793154 26929 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d8b83e1b70d41548334b38ed94234cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d8b83e1b70d41548334b38ed94234cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d8b83e1b70d41548334b38ed94234cd" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 45707 } } }
I20260925 22:40:10.793308 26930 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d8b83e1b70d41548334b38ed94234cd [sys.catalog]: This master's current role is: LEADER
I20260925 22:40:10.793329 26929 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d8b83e1b70d41548334b38ed94234cd [sys.catalog]: This master's current role is: LEADER
I20260925 22:40:10.793808 26937 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:40:10.797325 26937 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:40:10.804419 26937 catalog_manager.cc:1347] Generated new cluster ID: e493ef1db5ac4b9b9ef3d9355d4019fb
I20260925 22:40:10.804518 26937 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:40:10.811666 26937 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:40:10.813099 26937 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:40:10.819773 26937 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d8b83e1b70d41548334b38ed94234cd: Generated new TSK 0
I20260925 22:40:10.820797 26937 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:40:10.835721 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790375752797504-7131-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790375752797504-7131-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790375752797504-7131-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.193:0
--local_ip_for_outbound_sockets=127.6.246.193
--webserver_interface=127.6.246.193
--webserver_port=0
--tserver_master_addrs=127.6.246.254:45707
--builtin_ntp_servers=127.6.246.212:45947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 22:40:10.965833 26947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:40:10.966164 26947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:40:10.966207 26947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:40:10.970325 26947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:40:10.970387 26947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:40:10.970448 26947 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.193
I20260925 22:40:10.976007 26947 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:45947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790375752797504-7131-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790375752797504-7131-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.246.193
--webserver_port=0
--tserver_master_addrs=127.6.246.254:45707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26947
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.193
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790375752797504-7131-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:40:10.977430 26947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:40:10.979120 26947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:40:10.990250 26955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:10.990314 26953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:10.990259 26952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:40:10.990461 26947 server_base.cc:1034] running on GCE node
I20260925 22:40:10.991129 26947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:40:10.991879 26947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:40:10.993065 26947 hybrid_clock.cc:648] HybridClock initialized: now 1790376010993026 us; error 51 us; skew 500 ppm
I20260925 22:40:10.995292 26947 webserver.cc:460] Webserver started at http://127.6.246.193:38687/ using document root <none> and password file <none>
I20260925 22:40:10.996018 26947 fs_manager.cc:359] Metadata directory not provided
I20260925 22:40:10.996089 26947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:40:10.996374 26947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:40:10.998931 26947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790375752797504-7131-0/minicluster-data/ts-0/data/instance:
uuid: "b3da6020f95e41de93b36fd3ddacf34a"
format_stamp: "Formatted at 2026-09-25 22:40:10 on dist-test-slave-wjdr"
server_key: "4fa365639e5329efb40c7afd37fa61c6"
server_key_iv: "8fce88bbd4464268b5b5c4cf2663226d"
server_key_version: "encryptionkey@0"
I20260925 22:40:10.999620 26947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790375752797504-7131-0/minicluster-data/ts-0/wal/instance:
uuid: "b3da6020f95e41de93b36fd3ddacf34a"
format_stamp: "Formatted at 2026-09-25 22:40:10 on dist-test-slave-wjdr"
server_key: "4fa365639e5329efb40c7afd37fa61c6"
server_key_iv: "8fce88bbd4464268b5b5c4cf2663226d"
server_key_version: "encryptionkey@0"
I20260925 22:40:11.004113 26947 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 22:40:11.007016 26961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:11.008714 26947 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:40:11.008874 26947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790375752797504-7131-0/minicluster-data/ts-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790375752797504-7131-0/minicluster-data/ts-0/wal
uuid: "b3da6020f95e41de93b36fd3ddacf34a"
format_stamp: "Formatted at 2026-09-25 22:40:10 on dist-test-slave-wjdr"
server_key: "4fa365639e5329efb40c7afd37fa61c6"
server_key_iv: "8fce88bbd4464268b5b5c4cf2663226d"
server_key_version: "encryptionkey@0"
I20260925 22:40:11.008999 26947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790375752797504-7131-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790375752797504-7131-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790375752797504-7131-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:40:11.028261 26947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:40:11.029232 26947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:40:11.029474 26947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:40:11.030236 26947 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:40:11.031414 26947 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:40:11.031463 26947 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:11.031545 26947 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:40:11.031586 26947 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:11.042271 26947 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.193:36697
I20260925 22:40:11.042292 27074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.193:36697 every 8 connection(s)
I20260925 22:40:11.043491 26947 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790375752797504-7131-0/minicluster-data/ts-0/data/info.pb
I20260925 22:40:11.044229 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 26947
I20260925 22:40:11.044377 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790375752797504-7131-0/minicluster-data/ts-0/wal/instance
I20260925 22:40:11.044835 7131 external_mini_cluster.cc:1428] Setting key 65894f49b47903c59e2650d71dd04bec
I20260925 22:40:11.047549 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790375752797504-7131-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790375752797504-7131-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790375752797504-7131-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.194:0
--local_ip_for_outbound_sockets=127.6.246.194
--webserver_interface=127.6.246.194
--webserver_port=0
--tserver_master_addrs=127.6.246.254:45707
--builtin_ntp_servers=127.6.246.212:45947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 22:40:11.055817 26926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:11.045959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.193:44811 (local address 127.6.246.254:45707)
0925 22:40:11.046550 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:11.046556 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:11.047779 (+ 1223us) server_negotiation.cc:407] Connection header received
0925 22:40:11.049554 (+ 1775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:11.049559 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:11.049636 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:11.049747 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:11.050849 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:11.051776 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:11.053154 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:11.053591 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:11.053774 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:11.055116 (+ 1342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:11.055122 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:11.055126 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:40:11.055150 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:11.055183 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:40:11.055200 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:11.055295 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:11.055462 (+ 167us) server_negotiation.cc:299] Negotiation successful
0925 22:40:11.055554 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491}
I20260925 22:40:11.056929 27077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:11.046589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:45707 (local address 127.6.246.193:44811)
0925 22:40:11.047511 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:11.047574 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:11.049302 (+ 1728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:11.049938 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:11.049945 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:11.050045 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:40:11.050677 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:11.050691 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:11.051917 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:11.051919 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:11.052999 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:11.053008 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:11.053796 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:11.053799 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:11.053957 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:11.054782 (+ 825us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:40:11.054784 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:40:11.054836 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:40:11.054919 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:40:11.055300 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:40:11.055307 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:40:11.055312 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 22:40:11.055425 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 22:40:11.055718 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":178,"threads_started":1}
I20260925 22:40:11.058532 27075 heartbeater.cc:347] Connected to a master server at 127.6.246.254:45707
I20260925 22:40:11.058863 27075 heartbeater.cc:464] Registering TS with master...
I20260925 22:40:11.059716 27075 heartbeater.cc:511] Master 127.6.246.254:45707 requested a full tablet report, sending...
I20260925 22:40:11.061828 26887 ts_manager.cc:194] Registered new tserver with Master: b3da6020f95e41de93b36fd3ddacf34a (127.6.246.193:36697)
I20260925 22:40:11.063246 26887 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.246.193:44811
W20260925 22:40:11.181463 27078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:40:11.181802 27078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:40:11.181843 27078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:40:11.186300 27078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:40:11.186412 27078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:40:11.186514 27078 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.194
I20260925 22:40:11.191555 27078 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:45947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790375752797504-7131-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790375752797504-7131-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.246.194
--webserver_port=0
--tserver_master_addrs=127.6.246.254:45707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27078
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.194
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790375752797504-7131-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:40:11.193137 27078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:40:11.194972 27078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:40:11.206444 27086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:11.206444 27083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:11.206465 27084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:40:11.206935 27078 server_base.cc:1034] running on GCE node
I20260925 22:40:11.207450 27078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:40:11.208276 27078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:40:11.209491 27078 hybrid_clock.cc:648] HybridClock initialized: now 1790376011209483 us; error 45 us; skew 500 ppm
I20260925 22:40:11.211599 27078 webserver.cc:460] Webserver started at http://127.6.246.194:44621/ using document root <none> and password file <none>
I20260925 22:40:11.212313 27078 fs_manager.cc:359] Metadata directory not provided
I20260925 22:40:11.212423 27078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:40:11.212841 27078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:40:11.215636 27078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790375752797504-7131-0/minicluster-data/ts-1/data/instance:
uuid: "be2b33993ea44d98bfe1d543fa68c0d1"
format_stamp: "Formatted at 2026-09-25 22:40:11 on dist-test-slave-wjdr"
server_key: "02ed2c7c04880f6d4ab6d605906defe3"
server_key_iv: "1afde7a5d514a59693f6744e7a344987"
server_key_version: "encryptionkey@0"
I20260925 22:40:11.216328 27078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790375752797504-7131-0/minicluster-data/ts-1/wal/instance:
uuid: "be2b33993ea44d98bfe1d543fa68c0d1"
format_stamp: "Formatted at 2026-09-25 22:40:11 on dist-test-slave-wjdr"
server_key: "02ed2c7c04880f6d4ab6d605906defe3"
server_key_iv: "1afde7a5d514a59693f6744e7a344987"
server_key_version: "encryptionkey@0"
I20260925 22:40:11.220698 27078 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:40:11.223500 27092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:11.225068 27078 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:40:11.225226 27078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790375752797504-7131-0/minicluster-data/ts-1/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790375752797504-7131-0/minicluster-data/ts-1/wal
uuid: "be2b33993ea44d98bfe1d543fa68c0d1"
format_stamp: "Formatted at 2026-09-25 22:40:11 on dist-test-slave-wjdr"
server_key: "02ed2c7c04880f6d4ab6d605906defe3"
server_key_iv: "1afde7a5d514a59693f6744e7a344987"
server_key_version: "encryptionkey@0"
I20260925 22:40:11.225360 27078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790375752797504-7131-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790375752797504-7131-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790375752797504-7131-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:40:11.244692 27078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:40:11.245555 27078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:40:11.245798 27078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:40:11.246541 27078 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:40:11.247722 27078 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:40:11.247810 27078 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:11.247890 27078 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:40:11.247951 27078 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:11.259354 27078 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.194:44259
I20260925 22:40:11.259374 27205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.194:44259 every 8 connection(s)
I20260925 22:40:11.260625 27078 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790375752797504-7131-0/minicluster-data/ts-1/data/info.pb
I20260925 22:40:11.266584 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 27078
I20260925 22:40:11.266752 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790375752797504-7131-0/minicluster-data/ts-1/wal/instance
I20260925 22:40:11.267112 7131 external_mini_cluster.cc:1428] Setting key 28c706562ea22547609cfc2fba47c5c9
I20260925 22:40:11.269585 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790375752797504-7131-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790375752797504-7131-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790375752797504-7131-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.246.195:0
--local_ip_for_outbound_sockets=127.6.246.195
--webserver_interface=127.6.246.195
--webserver_port=0
--tserver_master_addrs=127.6.246.254:45707
--builtin_ntp_servers=127.6.246.212:45947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 22:40:11.272487 26926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:11.262875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.194:47399 (local address 127.6.246.254:45707)
0925 22:40:11.263148 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:11.263152 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:11.264346 (+ 1194us) server_negotiation.cc:407] Connection header received
0925 22:40:11.266079 (+ 1733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:11.266083 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:11.266158 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:11.266314 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:11.267508 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:11.268279 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:11.269894 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:11.270128 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:11.270276 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:11.271660 (+ 1384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:11.271667 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:11.271673 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:40:11.271694 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:11.271729 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:40:11.271745 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:11.271843 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:11.272143 (+ 300us) server_negotiation.cc:299] Negotiation successful
0925 22:40:11.272232 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158}
I20260925 22:40:11.273445 27208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:11.263315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:45707 (local address 127.6.246.194:47399)
0925 22:40:11.264151 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:11.264202 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:11.265806 (+ 1604us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:11.266509 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:11.266518 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:11.266611 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:40:11.267325 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:11.267341 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:11.268438 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:11.268442 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:11.269646 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:11.269657 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:11.270326 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:11.270330 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:11.270521 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:11.271346 (+ 825us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:40:11.271348 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:40:11.271394 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:40:11.271469 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:40:11.271875 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:40:11.271901 (+ 26us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:40:11.271906 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 22:40:11.272106 (+ 200us) client_negotiation.cc:240] Negotiation successful
0925 22:40:11.272392 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":165,"threads_started":1}
I20260925 22:40:11.274904 27206 heartbeater.cc:347] Connected to a master server at 127.6.246.254:45707
I20260925 22:40:11.275170 27206 heartbeater.cc:464] Registering TS with master...
I20260925 22:40:11.275805 27206 heartbeater.cc:511] Master 127.6.246.254:45707 requested a full tablet report, sending...
I20260925 22:40:11.277136 26887 ts_manager.cc:194] Registered new tserver with Master: be2b33993ea44d98bfe1d543fa68c0d1 (127.6.246.194:44259)
I20260925 22:40:11.277806 26887 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.246.194:47399
W20260925 22:40:11.400897 27209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:40:11.401279 27209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:40:11.401358 27209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:40:11.406324 27209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:40:11.406416 27209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:40:11.406481 27209 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.246.195
I20260925 22:40:11.412263 27209 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:45947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790375752797504-7131-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790375752797504-7131-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.246.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.246.195
--webserver_port=0
--tserver_master_addrs=127.6.246.254:45707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.246.195
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790375752797504-7131-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:40:11.414047 27209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:40:11.415869 27209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:40:11.427323 27217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:40:11.427381 27209 server_base.cc:1034] running on GCE node
W20260925 22:40:11.427304 27214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:11.427325 27215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:40:11.427999 27209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:40:11.428877 27209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:40:11.430094 27209 hybrid_clock.cc:648] HybridClock initialized: now 1790376011430072 us; error 35 us; skew 500 ppm
I20260925 22:40:11.432199 27209 webserver.cc:460] Webserver started at http://127.6.246.195:45435/ using document root <none> and password file <none>
I20260925 22:40:11.433015 27209 fs_manager.cc:359] Metadata directory not provided
I20260925 22:40:11.433136 27209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:40:11.433385 27209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:40:11.435794 27209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790375752797504-7131-0/minicluster-data/ts-2/data/instance:
uuid: "e2519ae6cfb34aab9531e1c4b5a68fe6"
format_stamp: "Formatted at 2026-09-25 22:40:11 on dist-test-slave-wjdr"
server_key: "2f5fce07536f9b5378c20ea00968b346"
server_key_iv: "759268086b7eb0932a398d5484b28561"
server_key_version: "encryptionkey@0"
I20260925 22:40:11.436429 27209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790375752797504-7131-0/minicluster-data/ts-2/wal/instance:
uuid: "e2519ae6cfb34aab9531e1c4b5a68fe6"
format_stamp: "Formatted at 2026-09-25 22:40:11 on dist-test-slave-wjdr"
server_key: "2f5fce07536f9b5378c20ea00968b346"
server_key_iv: "759268086b7eb0932a398d5484b28561"
server_key_version: "encryptionkey@0"
I20260925 22:40:11.440817 27209 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:40:11.443589 27223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:11.445143 27209 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:40:11.445324 27209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790375752797504-7131-0/minicluster-data/ts-2/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790375752797504-7131-0/minicluster-data/ts-2/wal
uuid: "e2519ae6cfb34aab9531e1c4b5a68fe6"
format_stamp: "Formatted at 2026-09-25 22:40:11 on dist-test-slave-wjdr"
server_key: "2f5fce07536f9b5378c20ea00968b346"
server_key_iv: "759268086b7eb0932a398d5484b28561"
server_key_version: "encryptionkey@0"
I20260925 22:40:11.445456 27209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790375752797504-7131-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790375752797504-7131-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790375752797504-7131-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:40:11.467758 27209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:40:11.468842 27209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:40:11.469149 27209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:40:11.470068 27209 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:40:11.471374 27209 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:40:11.471459 27209 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:11.471549 27209 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:40:11.471604 27209 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:11.482226 27209 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.195:45481
I20260925 22:40:11.482244 27336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.195:45481 every 8 connection(s)
I20260925 22:40:11.483378 27209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790375752797504-7131-0/minicluster-data/ts-2/data/info.pb
I20260925 22:40:11.488574 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 27209
I20260925 22:40:11.488729 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790375752797504-7131-0/minicluster-data/ts-2/wal/instance
I20260925 22:40:11.489076 7131 external_mini_cluster.cc:1428] Setting key 0575e42d7945b17952e8248a2342996c
I20260925 22:40:11.494953 26926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:11.485989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.195:34435 (local address 127.6.246.254:45707)
0925 22:40:11.486215 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:11.486219 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:11.487473 (+ 1254us) server_negotiation.cc:407] Connection header received
0925 22:40:11.489290 (+ 1817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:11.489294 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:11.489367 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:11.489478 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:11.490703 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:11.491345 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:11.492800 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:11.493073 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:11.493178 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:11.494472 (+ 1294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:11.494475 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:11.494478 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:40:11.494491 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:11.494522 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:40:11.494530 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:11.494595 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:11.494750 (+ 155us) server_negotiation.cc:299] Negotiation successful
0925 22:40:11.494812 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260925 22:40:11.495973 27339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:11.486461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:45707 (local address 127.6.246.195:34435)
0925 22:40:11.487279 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:11.487326 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:11.489015 (+ 1689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:11.489716 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:11.489731 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:11.489827 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:40:11.490555 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:11.490581 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:11.491494 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:11.491498 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:11.492640 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:11.492650 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:11.493229 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:11.493232 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:11.493392 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:11.494182 (+ 790us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:40:11.494184 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:40:11.494231 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:40:11.494305 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:40:11.494615 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:40:11.494622 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:40:11.494627 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 22:40:11.494733 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 22:40:11.494997 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":144,"threads_started":1}
I20260925 22:40:11.497252 27337 heartbeater.cc:347] Connected to a master server at 127.6.246.254:45707
I20260925 22:40:11.497499 27337 heartbeater.cc:464] Registering TS with master...
I20260925 22:40:11.498083 27337 heartbeater.cc:511] Master 127.6.246.254:45707 requested a full tablet report, sending...
I20260925 22:40:11.499296 26887 ts_manager.cc:194] Registered new tserver with Master: e2519ae6cfb34aab9531e1c4b5a68fe6 (127.6.246.195:45481)
I20260925 22:40:11.499900 26887 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.246.195:34435
I20260925 22:40:11.503610 7131 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:40:11.510038 26926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:11.506034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51384 (local address 127.6.246.254:45707)
0925 22:40:11.506231 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:11.506235 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:11.506415 (+ 180us) server_negotiation.cc:407] Connection header received
0925 22:40:11.506516 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:11.506522 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:11.506596 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:11.506683 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:11.506946 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:11.507481 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:11.508313 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:11.508602 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:11.508716 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:11.509557 (+ 841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:11.509565 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:11.509567 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:40:11.509579 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:11.509599 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:40:11.509607 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:11.509691 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:11.509815 (+ 124us) server_negotiation.cc:299] Negotiation successful
0925 22:40:11.509872 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260925 22:40:11.513257 26887 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51384:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:40:11.516968 26887 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:40:11.534070 27350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:11.526974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.195:45481 (local address 127.0.0.1:40322)
0925 22:40:11.528632 (+ 1658us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:11.528644 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:11.528736 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:11.529407 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:11.529410 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:11.529427 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:40:11.529515 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:11.529520 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:11.531096 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:11.531099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:11.532062 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:11.532071 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:11.533202 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:11.533205 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:11.533337 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:11.533563 (+ 226us) client_negotiation.cc:769] Sending connection context
0925 22:40:11.533684 (+ 121us) client_negotiation.cc:240] Negotiation successful
0925 22:40:11.533861 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1594,"mutex_wait_us":392,"thread_start_us":103,"threads_started":1}
I20260925 22:40:11.534442 27348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:11.527201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.193:36697 (local address 127.0.0.1:48210)
0925 22:40:11.528299 (+ 1098us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:11.528315 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:11.528407 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:11.529172 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:11.529181 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:11.529208 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:40:11.529403 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:11.529410 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:11.531058 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:11.531062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:11.532628 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:11.532635 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:11.533959 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:11.533962 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:11.534082 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:11.534108 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 22:40:11.534190 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 22:40:11.534276 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":987,"mutex_wait_us":262,"spinlock_wait_cycles":2944,"thread_start_us":109,"threads_started":1}
I20260925 22:40:11.534603 27351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:11.527316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40322 (local address 127.6.246.195:45481)
0925 22:40:11.528644 (+ 1328us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:11.528655 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:11.528713 (+ 58us) server_negotiation.cc:407] Connection header received
0925 22:40:11.528874 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:11.528881 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:11.529081 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:11.529248 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:40:11.529652 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:11.530928 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:11.532211 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:11.533062 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:11.533180 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:11.533416 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:11.534311 (+ 895us) server_negotiation.cc:299] Negotiation successful
0925 22:40:11.534432 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1208,"thread_start_us":111,"threads_started":1}
I20260925 22:40:11.535007 27353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:11.527914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48210 (local address 127.6.246.193:36697)
0925 22:40:11.528483 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:11.528491 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:11.528518 (+ 27us) server_negotiation.cc:407] Connection header received
0925 22:40:11.528614 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:11.528620 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:11.528872 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:11.529520 (+ 648us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:40:11.529707 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:11.530927 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:11.532758 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:11.533580 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:11.533687 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:11.533898 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:11.534745 (+ 847us) server_negotiation.cc:299] Negotiation successful
0925 22:40:11.534855 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":104,"threads_started":1}
I20260925 22:40:11.535598 27349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:11.526974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.194:44259 (local address 127.0.0.1:35856)
0925 22:40:11.527926 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:11.527989 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:11.528140 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:11.529238 (+ 1098us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:11.529241 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:11.529270 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:40:11.529391 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:11.529399 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:11.530927 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:11.530929 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:11.532624 (+ 1695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:11.532635 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:11.533902 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:11.533905 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:11.533994 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:11.534015 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 22:40:11.534086 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 22:40:11.534676 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":736,"mutex_wait_us":113,"spinlock_wait_cycles":768,"thread_start_us":93,"threads_started":1}
I20260925 22:40:11.536137 27352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:11.527759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35856 (local address 127.6.246.194:44259)
0925 22:40:11.528439 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:11.528456 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:11.528516 (+ 60us) server_negotiation.cc:407] Connection header received
0925 22:40:11.528625 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:11.528631 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:11.528866 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:11.529079 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:40:11.529634 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:11.530790 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:11.532789 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:11.533480 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:11.533585 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:11.533828 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:11.535905 (+ 2077us) server_negotiation.cc:299] Negotiation successful
0925 22:40:11.535998 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":152,"threads_started":1}
I20260925 22:40:11.538280 27271 tablet_service.cc:1467] Processing CreateTablet for tablet 4e11a319ac7f49a19d562c5cbcbfe90d (DEFAULT_TABLE table=test-table [id=b4cf7642a4ee411ebfc87f35f62c62b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:40:11.538277 27009 tablet_service.cc:1467] Processing CreateTablet for tablet 4e11a319ac7f49a19d562c5cbcbfe90d (DEFAULT_TABLE table=test-table [id=b4cf7642a4ee411ebfc87f35f62c62b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:40:11.538811 27140 tablet_service.cc:1467] Processing CreateTablet for tablet 4e11a319ac7f49a19d562c5cbcbfe90d (DEFAULT_TABLE table=test-table [id=b4cf7642a4ee411ebfc87f35f62c62b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:40:11.539636 27271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e11a319ac7f49a19d562c5cbcbfe90d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:40:11.539635 27009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e11a319ac7f49a19d562c5cbcbfe90d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:40:11.539824 27140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e11a319ac7f49a19d562c5cbcbfe90d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:40:11.547260 27354 tablet_bootstrap.cc:492] T 4e11a319ac7f49a19d562c5cbcbfe90d P be2b33993ea44d98bfe1d543fa68c0d1: Bootstrap starting.
I20260925 22:40:11.548255 27355 tablet_bootstrap.cc:492] T 4e11a319ac7f49a19d562c5cbcbfe90d P b3da6020f95e41de93b36fd3ddacf34a: Bootstrap starting.
I20260925 22:40:11.550599 27354 tablet_bootstrap.cc:654] T 4e11a319ac7f49a19d562c5cbcbfe90d P be2b33993ea44d98bfe1d543fa68c0d1: Neither blocks nor log segments found. Creating new log.
I20260925 22:40:11.550632 27355 tablet_bootstrap.cc:654] T 4e11a319ac7f49a19d562c5cbcbfe90d P b3da6020f95e41de93b36fd3ddacf34a: Neither blocks nor log segments found. Creating new log.
I20260925 22:40:11.551476 27354 log.cc:824] T 4e11a319ac7f49a19d562c5cbcbfe90d P be2b33993ea44d98bfe1d543fa68c0d1: Log is configured to *not* fsync() on all Append() calls
I20260925 22:40:11.551473 27355 log.cc:824] T 4e11a319ac7f49a19d562c5cbcbfe90d P b3da6020f95e41de93b36fd3ddacf34a: Log is configured to *not* fsync() on all Append() calls
I20260925 22:40:11.552307 27356 tablet_bootstrap.cc:492] T 4e11a319ac7f49a19d562c5cbcbfe90d P e2519ae6cfb34aab9531e1c4b5a68fe6: Bootstrap starting.
I20260925 22:40:11.554054 27354 tablet_bootstrap.cc:492] T 4e11a319ac7f49a19d562c5cbcbfe90d P be2b33993ea44d98bfe1d543fa68c0d1: No bootstrap required, opened a new log
I20260925 22:40:11.554280 27356 tablet_bootstrap.cc:654] T 4e11a319ac7f49a19d562c5cbcbfe90d P e2519ae6cfb34aab9531e1c4b5a68fe6: Neither blocks nor log segments found. Creating new log.
I20260925 22:40:11.554296 27354 ts_tablet_manager.cc:1397] T 4e11a319ac7f49a19d562c5cbcbfe90d P be2b33993ea44d98bfe1d543fa68c0d1: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260925 22:40:11.555123 27356 log.cc:824] T 4e11a319ac7f49a19d562c5cbcbfe90d P e2519ae6cfb34aab9531e1c4b5a68fe6: Log is configured to *not* fsync() on all Append() calls
I20260925 22:40:11.556591 27355 tablet_bootstrap.cc:492] T 4e11a319ac7f49a19d562c5cbcbfe90d P b3da6020f95e41de93b36fd3ddacf34a: No bootstrap required, opened a new log
I20260925 22:40:11.556807 27355 ts_tablet_manager.cc:1397] T 4e11a319ac7f49a19d562c5cbcbfe90d P b3da6020f95e41de93b36fd3ddacf34a: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260925 22:40:11.557065 27356 tablet_bootstrap.cc:492] T 4e11a319ac7f49a19d562c5cbcbfe90d P e2519ae6cfb34aab9531e1c4b5a68fe6: No bootstrap required, opened a new log
I20260925 22:40:11.557265 27356 ts_tablet_manager.cc:1397] T 4e11a319ac7f49a19d562c5cbcbfe90d P e2519ae6cfb34aab9531e1c4b5a68fe6: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 22:40:11.558703 27354 raft_consensus.cc:348] T 4e11a319ac7f49a19d562c5cbcbfe90d P be2b33993ea44d98bfe1d543fa68c0d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2519ae6cfb34aab9531e1c4b5a68fe6" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 45481 } } peers { permanent_uuid: "be2b33993ea44d98bfe1d543fa68c0d1" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 44259 } } peers { permanent_uuid: "b3da6020f95e41de93b36fd3ddacf34a" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 36697 } }
I20260925 22:40:11.559005 27354 raft_consensus.cc:374] T 4e11a319ac7f49a19d562c5cbcbfe90d P be2b33993ea44d98bfe1d543fa68c0d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:40:11.559064 27354 raft_consensus.cc:729] T 4e11a319ac7f49a19d562c5cbcbfe90d P be2b33993ea44d98bfe1d543fa68c0d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be2b33993ea44d98bfe1d543fa68c0d1, State: Initialized, Role: FOLLOWER
I20260925 22:40:11.559621 27354 consensus_queue.cc:260] T 4e11a319ac7f49a19d562c5cbcbfe90d P be2b33993ea44d98bfe1d543fa68c0d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2519ae6cfb34aab9531e1c4b5a68fe6" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 45481 } } peers { permanent_uuid: "be2b33993ea44d98bfe1d543fa68c0d1" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 44259 } } peers { permanent_uuid: "b3da6020f95e41de93b36fd3ddacf34a" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 36697 } }
I20260925 22:40:11.560427 27206 heartbeater.cc:503] Master 127.6.246.254:45707 was elected leader, sending a full tablet report...
I20260925 22:40:11.560798 27354 ts_tablet_manager.cc:1428] T 4e11a319ac7f49a19d562c5cbcbfe90d P be2b33993ea44d98bfe1d543fa68c0d1: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260925 22:40:11.561149 27356 raft_consensus.cc:348] T 4e11a319ac7f49a19d562c5cbcbfe90d P e2519ae6cfb34aab9531e1c4b5a68fe6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2519ae6cfb34aab9531e1c4b5a68fe6" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 45481 } } peers { permanent_uuid: "be2b33993ea44d98bfe1d543fa68c0d1" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 44259 } } peers { permanent_uuid: "b3da6020f95e41de93b36fd3ddacf34a" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 36697 } }
I20260925 22:40:11.561216 27355 raft_consensus.cc:348] T 4e11a319ac7f49a19d562c5cbcbfe90d P b3da6020f95e41de93b36fd3ddacf34a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2519ae6cfb34aab9531e1c4b5a68fe6" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 45481 } } peers { permanent_uuid: "be2b33993ea44d98bfe1d543fa68c0d1" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 44259 } } peers { permanent_uuid: "b3da6020f95e41de93b36fd3ddacf34a" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 36697 } }
I20260925 22:40:11.561394 27356 raft_consensus.cc:374] T 4e11a319ac7f49a19d562c5cbcbfe90d P e2519ae6cfb34aab9531e1c4b5a68fe6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:40:11.561406 27355 raft_consensus.cc:374] T 4e11a319ac7f49a19d562c5cbcbfe90d P b3da6020f95e41de93b36fd3ddacf34a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:40:11.561437 27355 raft_consensus.cc:729] T 4e11a319ac7f49a19d562c5cbcbfe90d P b3da6020f95e41de93b36fd3ddacf34a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3da6020f95e41de93b36fd3ddacf34a, State: Initialized, Role: FOLLOWER
I20260925 22:40:11.561450 27356 raft_consensus.cc:729] T 4e11a319ac7f49a19d562c5cbcbfe90d P e2519ae6cfb34aab9531e1c4b5a68fe6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2519ae6cfb34aab9531e1c4b5a68fe6, State: Initialized, Role: FOLLOWER
I20260925 22:40:11.561846 27355 consensus_queue.cc:260] T 4e11a319ac7f49a19d562c5cbcbfe90d P b3da6020f95e41de93b36fd3ddacf34a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2519ae6cfb34aab9531e1c4b5a68fe6" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 45481 } } peers { permanent_uuid: "be2b33993ea44d98bfe1d543fa68c0d1" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 44259 } } peers { permanent_uuid: "b3da6020f95e41de93b36fd3ddacf34a" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 36697 } }
I20260925 22:40:11.561882 27356 consensus_queue.cc:260] T 4e11a319ac7f49a19d562c5cbcbfe90d P e2519ae6cfb34aab9531e1c4b5a68fe6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2519ae6cfb34aab9531e1c4b5a68fe6" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 45481 } } peers { permanent_uuid: "be2b33993ea44d98bfe1d543fa68c0d1" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 44259 } } peers { permanent_uuid: "b3da6020f95e41de93b36fd3ddacf34a" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 36697 } }
I20260925 22:40:11.562494 27075 heartbeater.cc:503] Master 127.6.246.254:45707 was elected leader, sending a full tablet report...
I20260925 22:40:11.562819 27356 ts_tablet_manager.cc:1428] T 4e11a319ac7f49a19d562c5cbcbfe90d P e2519ae6cfb34aab9531e1c4b5a68fe6: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260925 22:40:11.562851 27355 ts_tablet_manager.cc:1428] T 4e11a319ac7f49a19d562c5cbcbfe90d P b3da6020f95e41de93b36fd3ddacf34a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 22:40:11.563153 27337 heartbeater.cc:503] Master 127.6.246.254:45707 was elected leader, sending a full tablet report...
I20260925 22:40:11.643399 27362 raft_consensus.cc:482] T 4e11a319ac7f49a19d562c5cbcbfe90d P e2519ae6cfb34aab9531e1c4b5a68fe6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:40:11.643797 27362 raft_consensus.cc:504] T 4e11a319ac7f49a19d562c5cbcbfe90d P e2519ae6cfb34aab9531e1c4b5a68fe6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2519ae6cfb34aab9531e1c4b5a68fe6" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 45481 } } peers { permanent_uuid: "be2b33993ea44d98bfe1d543fa68c0d1" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 44259 } } peers { permanent_uuid: "b3da6020f95e41de93b36fd3ddacf34a" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 36697 } }
I20260925 22:40:11.645493 27362 leader_election.cc:288] T 4e11a319ac7f49a19d562c5cbcbfe90d P e2519ae6cfb34aab9531e1c4b5a68fe6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be2b33993ea44d98bfe1d543fa68c0d1 (127.6.246.194:44259), b3da6020f95e41de93b36fd3ddacf34a (127.6.246.193:36697)
I20260925 22:40:11.650972 27353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:11.646029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.195:46695 (local address 127.6.246.193:36697)
0925 22:40:11.646204 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:11.646210 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:11.646434 (+ 224us) server_negotiation.cc:407] Connection header received
0925 22:40:11.646508 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:11.646513 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:11.646600 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:11.646674 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:40:11.647088 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:11.647856 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:11.649555 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:11.650240 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:11.650371 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:11.650430 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:11.650742 (+ 312us) server_negotiation.cc:299] Negotiation successful
0925 22:40:11.650818 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 22:40:11.650972 27339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:11.645789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.194:44259 (local address 127.6.246.195:50331)
0925 22:40:11.646044 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:11.646062 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:11.646222 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:11.646603 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:11.646607 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:11.646623 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:40:11.646765 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:11.646773 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:11.647896 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:11.647898 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:11.649424 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:11.649429 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:11.650370 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:11.650373 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:11.650547 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:11.650627 (+ 80us) client_negotiation.cc:769] Sending connection context
0925 22:40:11.650702 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 22:40:11.650793 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":212}
I20260925 22:40:11.650981 27363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:11.645862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.193:36697 (local address 127.6.246.195:46695)
0925 22:40:11.646323 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:11.646343 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:11.646429 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:11.646740 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:11.646743 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:11.646766 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:40:11.646934 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:11.646944 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:11.647964 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:11.647967 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:11.649417 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:11.649428 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:11.650370 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:11.650373 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:11.650573 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:11.650619 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 22:40:11.650702 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 22:40:11.650793 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"mutex_wait_us":117,"thread_start_us":223,"threads_started":1}
I20260925 22:40:11.650972 27352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:11.645881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.246.195:50331 (local address 127.6.246.194:44259)
0925 22:40:11.646053 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:11.646060 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:11.646255 (+ 195us) server_negotiation.cc:407] Connection header received
0925 22:40:11.646337 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:11.646339 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:11.646455 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:11.646568 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:40:11.646909 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:11.647784 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:11.649555 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:11.650240 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:11.650371 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:11.650435 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:11.650742 (+ 307us) server_negotiation.cc:299] Negotiation successful
0925 22:40:11.650826 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 22:40:11.651862 27029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e11a319ac7f49a19d562c5cbcbfe90d" candidate_uuid: "e2519ae6cfb34aab9531e1c4b5a68fe6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da6020f95e41de93b36fd3ddacf34a" is_pre_election: true
I20260925 22:40:11.651873 27160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e11a319ac7f49a19d562c5cbcbfe90d" candidate_uuid: "e2519ae6cfb34aab9531e1c4b5a68fe6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be2b33993ea44d98bfe1d543fa68c0d1" is_pre_election: true
I20260925 22:40:11.652227 27160 raft_consensus.cc:2454] T 4e11a319ac7f49a19d562c5cbcbfe90d P be2b33993ea44d98bfe1d543fa68c0d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2519ae6cfb34aab9531e1c4b5a68fe6 in term 0.
I20260925 22:40:11.652230 27029 raft_consensus.cc:2454] T 4e11a319ac7f49a19d562c5cbcbfe90d P b3da6020f95e41de93b36fd3ddacf34a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2519ae6cfb34aab9531e1c4b5a68fe6 in term 0.
I20260925 22:40:11.652776 27227 leader_election.cc:302] T 4e11a319ac7f49a19d562c5cbcbfe90d P e2519ae6cfb34aab9531e1c4b5a68fe6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be2b33993ea44d98bfe1d543fa68c0d1, e2519ae6cfb34aab9531e1c4b5a68fe6; no voters:
I20260925 22:40:11.653080 27362 raft_consensus.cc:2781] T 4e11a319ac7f49a19d562c5cbcbfe90d P e2519ae6cfb34aab9531e1c4b5a68fe6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:40:11.653169 27362 raft_consensus.cc:482] T 4e11a319ac7f49a19d562c5cbcbfe90d P e2519ae6cfb34aab9531e1c4b5a68fe6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:40:11.653211 27362 raft_consensus.cc:3037] T 4e11a319ac7f49a19d562c5cbcbfe90d P e2519ae6cfb34aab9531e1c4b5a68fe6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:40:11.654429 27362 raft_consensus.cc:504] T 4e11a319ac7f49a19d562c5cbcbfe90d P e2519ae6cfb34aab9531e1c4b5a68fe6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2519ae6cfb34aab9531e1c4b5a68fe6" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 45481 } } peers { permanent_uuid: "be2b33993ea44d98bfe1d543fa68c0d1" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 44259 } } peers { permanent_uuid: "b3da6020f95e41de93b36fd3ddacf34a" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 36697 } }
I20260925 22:40:11.654795 27362 leader_election.cc:288] T 4e11a319ac7f49a19d562c5cbcbfe90d P e2519ae6cfb34aab9531e1c4b5a68fe6 [CANDIDATE]: Term 1 election: Requested vote from peers be2b33993ea44d98bfe1d543fa68c0d1 (127.6.246.194:44259), b3da6020f95e41de93b36fd3ddacf34a (127.6.246.193:36697)
I20260925 22:40:11.655122 27160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e11a319ac7f49a19d562c5cbcbfe90d" candidate_uuid: "e2519ae6cfb34aab9531e1c4b5a68fe6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be2b33993ea44d98bfe1d543fa68c0d1"
I20260925 22:40:11.655200 27029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e11a319ac7f49a19d562c5cbcbfe90d" candidate_uuid: "e2519ae6cfb34aab9531e1c4b5a68fe6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da6020f95e41de93b36fd3ddacf34a"
I20260925 22:40:11.655269 27160 raft_consensus.cc:3037] T 4e11a319ac7f49a19d562c5cbcbfe90d P be2b33993ea44d98bfe1d543fa68c0d1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:40:11.655357 27029 raft_consensus.cc:3037] T 4e11a319ac7f49a19d562c5cbcbfe90d P b3da6020f95e41de93b36fd3ddacf34a [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:40:11.656538 27160 raft_consensus.cc:2454] T 4e11a319ac7f49a19d562c5cbcbfe90d P be2b33993ea44d98bfe1d543fa68c0d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2519ae6cfb34aab9531e1c4b5a68fe6 in term 1.
I20260925 22:40:11.656565 27029 raft_consensus.cc:2454] T 4e11a319ac7f49a19d562c5cbcbfe90d P b3da6020f95e41de93b36fd3ddacf34a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2519ae6cfb34aab9531e1c4b5a68fe6 in term 1.
I20260925 22:40:11.656869 27226 leader_election.cc:302] T 4e11a319ac7f49a19d562c5cbcbfe90d P e2519ae6cfb34aab9531e1c4b5a68fe6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3da6020f95e41de93b36fd3ddacf34a, e2519ae6cfb34aab9531e1c4b5a68fe6; no voters:
I20260925 22:40:11.657065 27362 raft_consensus.cc:2781] T 4e11a319ac7f49a19d562c5cbcbfe90d P e2519ae6cfb34aab9531e1c4b5a68fe6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:40:11.657334 27362 raft_consensus.cc:686] T 4e11a319ac7f49a19d562c5cbcbfe90d P e2519ae6cfb34aab9531e1c4b5a68fe6 [term 1 LEADER]: Becoming Leader. State: Replica: e2519ae6cfb34aab9531e1c4b5a68fe6, State: Running, Role: LEADER
I20260925 22:40:11.657714 27362 consensus_queue.cc:237] T 4e11a319ac7f49a19d562c5cbcbfe90d P e2519ae6cfb34aab9531e1c4b5a68fe6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2519ae6cfb34aab9531e1c4b5a68fe6" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 45481 } } peers { permanent_uuid: "be2b33993ea44d98bfe1d543fa68c0d1" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 44259 } } peers { permanent_uuid: "b3da6020f95e41de93b36fd3ddacf34a" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 36697 } }
I20260925 22:40:11.661578 26885 catalog_manager.cc:5495] T 4e11a319ac7f49a19d562c5cbcbfe90d P e2519ae6cfb34aab9531e1c4b5a68fe6 reported cstate change: term changed from 0 to 1, leader changed from <none> to e2519ae6cfb34aab9531e1c4b5a68fe6 (127.6.246.195). New cstate: current_term: 1 leader_uuid: "e2519ae6cfb34aab9531e1c4b5a68fe6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2519ae6cfb34aab9531e1c4b5a68fe6" member_type: VOTER last_known_addr { host: "127.6.246.195" port: 45481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be2b33993ea44d98bfe1d543fa68c0d1" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 44259 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3da6020f95e41de93b36fd3ddacf34a" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 36697 } health_report { overall_health: UNKNOWN } } }
I20260925 22:40:11.686769 27351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:11.682884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40332 (local address 127.6.246.195:45481)
0925 22:40:11.683051 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:11.683055 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:11.683070 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:40:11.683158 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:11.683161 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:11.683224 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:11.683326 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:40:11.683565 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:11.684155 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:11.685177 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:11.685421 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:11.685527 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:11.685790 (+ 263us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:40:11.686305 (+ 515us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:40:11.686407 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:11.686474 (+ 67us) server_negotiation.cc:299] Negotiation successful
0925 22:40:11.686537 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 22:40:11.694446 27029 raft_consensus.cc:1261] T 4e11a319ac7f49a19d562c5cbcbfe90d P b3da6020f95e41de93b36fd3ddacf34a [term 1 FOLLOWER]: Refusing update from remote peer e2519ae6cfb34aab9531e1c4b5a68fe6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:40:11.694446 27160 raft_consensus.cc:1261] T 4e11a319ac7f49a19d562c5cbcbfe90d P be2b33993ea44d98bfe1d543fa68c0d1 [term 1 FOLLOWER]: Refusing update from remote peer e2519ae6cfb34aab9531e1c4b5a68fe6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:40:11.695266 27362 consensus_queue.cc:1035] T 4e11a319ac7f49a19d562c5cbcbfe90d P e2519ae6cfb34aab9531e1c4b5a68fe6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da6020f95e41de93b36fd3ddacf34a" member_type: VOTER last_known_addr { host: "127.6.246.193" port: 36697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:40:11.695526 27364 consensus_queue.cc:1035] T 4e11a319ac7f49a19d562c5cbcbfe90d P e2519ae6cfb34aab9531e1c4b5a68fe6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be2b33993ea44d98bfe1d543fa68c0d1" member_type: VOTER last_known_addr { host: "127.6.246.194" port: 44259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:40:11.704779 27369 mvcc.cc:205] Tried to move back new op lower bound from 7333380143890108416 to 7333380143753211904. Current Snapshot: MvccSnapshot[applied={T|T < 7333380143890108416}]
I20260925 22:40:11.707185 27367 mvcc.cc:205] Tried to move back new op lower bound from 7333380143890108416 to 7333380143753211904. Current Snapshot: MvccSnapshot[applied={T|T < 7333380143890108416 or (T in {7333380143890108416})}]
I20260925 22:40:11.714733 26886 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51384:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:40:11.714951 26886 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51384:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:40:11.717836 26886 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8d8b83e1b70d41548334b38ed94234cd: Sending DeleteTablet for 3 replicas of tablet 4e11a319ac7f49a19d562c5cbcbfe90d
I20260925 22:40:11.718703 27271 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e11a319ac7f49a19d562c5cbcbfe90d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:40:11 UTC) from {username='slave'} at 127.0.0.1:40322
I20260925 22:40:11.718717 27140 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e11a319ac7f49a19d562c5cbcbfe90d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:40:11 UTC) from {username='slave'} at 127.0.0.1:35856
I20260925 22:40:11.719545 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 26947
I20260925 22:40:11.719591 27377 tablet_replica.cc:330] T 4e11a319ac7f49a19d562c5cbcbfe90d P e2519ae6cfb34aab9531e1c4b5a68fe6: stopping tablet replica
I20260925 22:40:11.719543 27009 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e11a319ac7f49a19d562c5cbcbfe90d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:40:11 UTC) from {username='slave'} at 127.0.0.1:48210
I20260925 22:40:11.720148 27378 tablet_replica.cc:330] T 4e11a319ac7f49a19d562c5cbcbfe90d P be2b33993ea44d98bfe1d543fa68c0d1: stopping tablet replica
I20260925 22:40:11.720558 27378 raft_consensus.cc:2229] T 4e11a319ac7f49a19d562c5cbcbfe90d P be2b33993ea44d98bfe1d543fa68c0d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:40:11.720918 27378 raft_consensus.cc:2258] T 4e11a319ac7f49a19d562c5cbcbfe90d P be2b33993ea44d98bfe1d543fa68c0d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:40:11.721338 27377 raft_consensus.cc:2229] T 4e11a319ac7f49a19d562c5cbcbfe90d P e2519ae6cfb34aab9531e1c4b5a68fe6 [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:40:11.721740 27377 raft_consensus.cc:2258] T 4e11a319ac7f49a19d562c5cbcbfe90d P e2519ae6cfb34aab9531e1c4b5a68fe6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:40:11.723531 27378 ts_tablet_manager.cc:1905] T 4e11a319ac7f49a19d562c5cbcbfe90d P be2b33993ea44d98bfe1d543fa68c0d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:40:11.723531 27377 ts_tablet_manager.cc:1905] T 4e11a319ac7f49a19d562c5cbcbfe90d P e2519ae6cfb34aab9531e1c4b5a68fe6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:40:11.726805 27378 ts_tablet_manager.cc:1918] T 4e11a319ac7f49a19d562c5cbcbfe90d P be2b33993ea44d98bfe1d543fa68c0d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:40:11.726915 27378 log.cc:1196] T 4e11a319ac7f49a19d562c5cbcbfe90d P be2b33993ea44d98bfe1d543fa68c0d1: Deleting WAL directory at /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790375752797504-7131-0/minicluster-data/ts-1/wal/wals/4e11a319ac7f49a19d562c5cbcbfe90d
I20260925 22:40:11.727257 27378 ts_tablet_manager.cc:1939] T 4e11a319ac7f49a19d562c5cbcbfe90d P be2b33993ea44d98bfe1d543fa68c0d1: Deleting consensus metadata
I20260925 22:40:11.727391 27377 ts_tablet_manager.cc:1918] T 4e11a319ac7f49a19d562c5cbcbfe90d P e2519ae6cfb34aab9531e1c4b5a68fe6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:40:11.727510 27377 log.cc:1196] T 4e11a319ac7f49a19d562c5cbcbfe90d P e2519ae6cfb34aab9531e1c4b5a68fe6: Deleting WAL directory at /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790375752797504-7131-0/minicluster-data/ts-2/wal/wals/4e11a319ac7f49a19d562c5cbcbfe90d
I20260925 22:40:11.727854 27377 ts_tablet_manager.cc:1939] T 4e11a319ac7f49a19d562c5cbcbfe90d P e2519ae6cfb34aab9531e1c4b5a68fe6: Deleting consensus metadata
I20260925 22:40:11.728886 26874 catalog_manager.cc:4841] TS be2b33993ea44d98bfe1d543fa68c0d1 (127.6.246.194:44259): tablet 4e11a319ac7f49a19d562c5cbcbfe90d (table test-table [id=b4cf7642a4ee411ebfc87f35f62c62b3]) successfully deleted
I20260925 22:40:11.729274 26871 catalog_manager.cc:4841] TS e2519ae6cfb34aab9531e1c4b5a68fe6 (127.6.246.195:45481): tablet 4e11a319ac7f49a19d562c5cbcbfe90d (table test-table [id=b4cf7642a4ee411ebfc87f35f62c62b3]) successfully deleted
W20260925 22:40:11.729516 26873 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.246.193:36697 (error 108)
I20260925 22:40:11.729703 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 27078
I20260925 22:40:11.731485 27348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:11.731088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.193:36697 (local address 127.0.0.1:48226)
0925 22:40:11.731255 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:11.731351 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.193:36697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260925 22:40:11.731938 26873 catalog_manager.cc:4584] TS b3da6020f95e41de93b36fd3ddacf34a (127.6.246.193:36697): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4e11a319ac7f49a19d562c5cbcbfe90d: Network error: Client connection negotiation failed: client connection to 127.6.246.193:36697: connect: Connection refused (error 111)
I20260925 22:40:11.736826 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 27209
I20260925 22:40:11.744180 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 26856
2026-09-25T22:40:11Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1209 ms)
[----------] 6 tests from EncryptionPolicyTest (11041 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790375752797504-7131-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:40:11 dist-test-slave-wjdr krb5kdc[27382](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:40:11 dist-test-slave-wjdr krb5kdc[27382](info): set up 2 sockets
Sep 25 22:40:11 dist-test-slave-wjdr krb5kdc[27382](info): commencing operation
krb5kdc: starting...
W20260925 22:40:12.796916 7131 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Sep 25 22:40:12 dist-test-slave-wjdr krb5kdc[27382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:40:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:40:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790375752797504-7131-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790375752797504-7131-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 22:40:12.923092 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790375752797504-7131-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790375752797504-7131-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790375752797504-7131-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34849
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790375752797504-7131-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34849
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:40:13.040979 27395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:40:13.041325 27395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:40:13.041404 27395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:40:13.046000 27395 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:40:13.046113 27395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:40:13.046165 27395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:40:13.046217 27395 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:40:13.046248 27395 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:40:13.051676 27395 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790375752797504-7131-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790375752797504-7131-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34849
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790375752797504-7131-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27395
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790375752797504-7131-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:40:13.052893 27395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:40:13.054035 27395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 22:40:13.063046 27395 server_base.cc:1034] running on GCE node
W20260925 22:40:13.062995 27400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:13.063010 27401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:13.062995 27403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:40:13.063890 27395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:40:13.065147 27395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:40:13.066313 27395 hybrid_clock.cc:648] HybridClock initialized: now 1790376013066283 us; error 47 us; skew 500 ppm
Sep 25 22:40:13 dist-test-slave-wjdr krb5kdc[27382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376013, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:40:13.069195 27395 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 22:40:13.070196 27395 webserver.cc:460] Webserver started at http://127.0.0.1:43157/ using document root <none> and password file <none>
I20260925 22:40:13.070806 27395 fs_manager.cc:359] Metadata directory not provided
I20260925 22:40:13.070917 27395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:40:13.071157 27395 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:40:13.073307 27395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790375752797504-7131-0/minicluster-data/master-0/data/instance:
uuid: "92798296b3ec408d857da8fc811a3805"
format_stamp: "Formatted at 2026-09-25 22:40:13 on dist-test-slave-wjdr"
server_key: "0d33c3e65a16caaee33018ab05d25641"
server_key_iv: "8aecc7d91bf092b74d4157cee90943d1"
server_key_version: "encryptionkey@0"
I20260925 22:40:13.073915 27395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790375752797504-7131-0/minicluster-data/master-0/wal/instance:
uuid: "92798296b3ec408d857da8fc811a3805"
format_stamp: "Formatted at 2026-09-25 22:40:13 on dist-test-slave-wjdr"
server_key: "0d33c3e65a16caaee33018ab05d25641"
server_key_iv: "8aecc7d91bf092b74d4157cee90943d1"
server_key_version: "encryptionkey@0"
I20260925 22:40:13.077770 27395 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:40:13.080547 27410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:13.081923 27395 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:40:13.082089 27395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790375752797504-7131-0/minicluster-data/master-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790375752797504-7131-0/minicluster-data/master-0/wal
uuid: "92798296b3ec408d857da8fc811a3805"
format_stamp: "Formatted at 2026-09-25 22:40:13 on dist-test-slave-wjdr"
server_key: "0d33c3e65a16caaee33018ab05d25641"
server_key_iv: "8aecc7d91bf092b74d4157cee90943d1"
server_key_version: "encryptionkey@0"
I20260925 22:40:13.082218 27395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790375752797504-7131-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790375752797504-7131-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790375752797504-7131-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:40:13.099550 27395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:40:13.103204 27395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:40:13.103442 27395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:40:13.111469 27395 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34849
I20260925 22:40:13.111557 27462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34849 every 8 connection(s)
I20260925 22:40:13.112706 27395 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
I20260925 22:40:13.115748 27463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:40:13.119624 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 27395
I20260925 22:40:13.119829 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790375752797504-7131-0/minicluster-data/master-0/wal/instance
I20260925 22:40:13.120148 7131 external_mini_cluster.cc:1428] Setting key 2719e9cc703ce084c91a32812ff87c6b
I20260925 22:40:13.122102 27463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92798296b3ec408d857da8fc811a3805: Bootstrap starting.
I20260925 22:40:13.124926 27463 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92798296b3ec408d857da8fc811a3805: Neither blocks nor log segments found. Creating new log.
I20260925 22:40:13.125842 27463 log.cc:824] T 00000000000000000000000000000000 P 92798296b3ec408d857da8fc811a3805: Log is configured to *not* fsync() on all Append() calls
I20260925 22:40:13.128193 27463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92798296b3ec408d857da8fc811a3805: No bootstrap required, opened a new log
Sep 25 22:40:13 dist-test-slave-wjdr krb5kdc[27382](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 22:40:13.133114 27463 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92798296b3ec408d857da8fc811a3805 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92798296b3ec408d857da8fc811a3805" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34849 } }
I20260925 22:40:13.133385 27463 raft_consensus.cc:374] T 00000000000000000000000000000000 P 92798296b3ec408d857da8fc811a3805 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:40:13.133471 27463 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92798296b3ec408d857da8fc811a3805 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92798296b3ec408d857da8fc811a3805, State: Initialized, Role: FOLLOWER
I20260925 22:40:13.133985 27463 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92798296b3ec408d857da8fc811a3805 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92798296b3ec408d857da8fc811a3805" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34849 } }
I20260925 22:40:13.134124 27463 raft_consensus.cc:388] T 00000000000000000000000000000000 P 92798296b3ec408d857da8fc811a3805 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:40:13.134219 27463 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92798296b3ec408d857da8fc811a3805 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:40:13.134336 27463 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92798296b3ec408d857da8fc811a3805 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:40:13.135399 27463 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92798296b3ec408d857da8fc811a3805 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92798296b3ec408d857da8fc811a3805" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34849 } }
I20260925 22:40:13.135790 27463 leader_election.cc:302] T 00000000000000000000000000000000 P 92798296b3ec408d857da8fc811a3805 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 92798296b3ec408d857da8fc811a3805; no voters:
I20260925 22:40:13.135841 27466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:13.121583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50057 (local address 127.0.0.1:34849)
0925 22:40:13.121999 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:13.122016 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:13.122050 (+ 34us) server_negotiation.cc:407] Connection header received
0925 22:40:13.122716 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:13.122734 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:13.123070 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:13.123363 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:13.124337 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:13.125484 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:13.126618 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:13.127129 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:13.127254 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:13.130192 (+ 2938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:13.130199 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:13.130216 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:13.130248 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:13.132350 (+ 2102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:13.132978 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:13.132983 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:13.132988 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:13.133103 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:13.133550 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:13.133553 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:13.133556 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:13.134132 (+ 576us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:13.134391 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:13.134793 (+ 402us) server_negotiation.cc:299] Negotiation successful
0925 22:40:13.135022 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":150,"threads_started":1}
I20260925 22:40:13.136121 27463 leader_election.cc:288] T 00000000000000000000000000000000 P 92798296b3ec408d857da8fc811a3805 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:40:13.136219 27468 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92798296b3ec408d857da8fc811a3805 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:40:13.136407 27468 raft_consensus.cc:686] T 00000000000000000000000000000000 P 92798296b3ec408d857da8fc811a3805 [term 1 LEADER]: Becoming Leader. State: Replica: 92798296b3ec408d857da8fc811a3805, State: Running, Role: LEADER
I20260925 22:40:13.136731 27468 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92798296b3ec408d857da8fc811a3805 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92798296b3ec408d857da8fc811a3805" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34849 } }
I20260925 22:40:13.137454 27463 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92798296b3ec408d857da8fc811a3805 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:40:13.139115 7131 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:40:13.139235 27470 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92798296b3ec408d857da8fc811a3805 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "92798296b3ec408d857da8fc811a3805" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92798296b3ec408d857da8fc811a3805" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34849 } } }
I20260925 22:40:13.139349 27470 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92798296b3ec408d857da8fc811a3805 [sys.catalog]: This master's current role is: LEADER
I20260925 22:40:13.139672 27469 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92798296b3ec408d857da8fc811a3805 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92798296b3ec408d857da8fc811a3805. Latest consensus state: current_term: 1 leader_uuid: "92798296b3ec408d857da8fc811a3805" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92798296b3ec408d857da8fc811a3805" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34849 } } }
I20260925 22:40:13.139765 27469 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92798296b3ec408d857da8fc811a3805 [sys.catalog]: This master's current role is: LEADER
W20260925 22:40:13.142737 27483 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 92798296b3ec408d857da8fc811a3805: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:40:13.142871 27483 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:40:13.142942 27482 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:40:13.143807 27482 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:40:13.150030 27482 catalog_manager.cc:1347] Generated new cluster ID: d94ed8f6f1d44a3fa38c22b0cadb98aa
I20260925 22:40:13.150099 27482 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:40:13.150877 27466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:13.141594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39233 (local address 127.0.0.1:34849)
0925 22:40:13.141705 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:13.141709 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:13.141938 (+ 229us) server_negotiation.cc:407] Connection header received
0925 22:40:13.142089 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:13.142092 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:13.142134 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:13.142226 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:13.142895 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:13.143617 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:13.144874 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:13.145210 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:13.145325 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:13.147085 (+ 1760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:13.147088 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:13.147091 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:13.147115 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:13.148846 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:13.149402 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:13.149405 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:13.149406 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:13.149495 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:13.149902 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:13.149905 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:13.149906 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:13.150113 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:13.150220 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:13.150592 (+ 372us) server_negotiation.cc:299] Negotiation successful
0925 22:40:13.150686 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 22:40:13.167567 27482 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:40:13.168637 27482 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:40:13.174293 27482 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 92798296b3ec408d857da8fc811a3805: Generated new TSK 0
I20260925 22:40:13.174997 27482 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:40:13.181195 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 27395
W20260925 22:40:13.187175 27380 connection.cc:535] client connection to 127.0.0.1:34849 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34849): Connection reset by peer (error 104)
2026-09-25T22:40:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1434 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790375752797504-7131-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:40:13 dist-test-slave-wjdr krb5kdc[27494](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:40:13 dist-test-slave-wjdr krb5kdc[27494](info): set up 2 sockets
Sep 25 22:40:13 dist-test-slave-wjdr krb5kdc[27494](info): commencing operation
krb5kdc: starting...
W20260925 22:40:14.229076 7131 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Sep 25 22:40:14 dist-test-slave-wjdr krb5kdc[27494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:40:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:40:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790375752797504-7131-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790375752797504-7131-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 22:40:14.363890 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790375752797504-7131-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790375752797504-7131-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790375752797504-7131-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44415
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790375752797504-7131-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44415
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:40:14.490751 27507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:40:14.491139 27507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:40:14.491227 27507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:40:14.495419 27507 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:40:14.495553 27507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:40:14.495589 27507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:40:14.495643 27507 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:40:14.495671 27507 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:40:14.501492 27507 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790375752797504-7131-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790375752797504-7131-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44415
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790375752797504-7131-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790375752797504-7131-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:40:14.502871 27507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:40:14.504136 27507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 22:40:14.513923 27507 server_base.cc:1034] running on GCE node
W20260925 22:40:14.513827 27515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:14.513834 27512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:14.513827 27513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:40:14.514861 27507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:40:14.516129 27507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:40:14.517329 27507 hybrid_clock.cc:648] HybridClock initialized: now 1790376014517315 us; error 29 us; skew 500 ppm
Sep 25 22:40:14 dist-test-slave-wjdr krb5kdc[27494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376014, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:40:14.525686 27507 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 22:40:14.526805 27507 webserver.cc:460] Webserver started at http://127.0.0.1:44529/ using document root <none> and password file <none>
I20260925 22:40:14.527490 27507 fs_manager.cc:359] Metadata directory not provided
I20260925 22:40:14.527590 27507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:40:14.527858 27507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:40:14.530364 27507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790375752797504-7131-0/minicluster-data/master-0/data/instance:
uuid: "0ec6a30510914149b2060b067c31b11e"
format_stamp: "Formatted at 2026-09-25 22:40:14 on dist-test-slave-wjdr"
server_key: "27fcfd8e33369c0b2d0ab91fb2aa9a73"
server_key_iv: "a6314afbc35c3085562a67d282e47ce6"
server_key_version: "encryptionkey@0"
I20260925 22:40:14.531010 27507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790375752797504-7131-0/minicluster-data/master-0/wal/instance:
uuid: "0ec6a30510914149b2060b067c31b11e"
format_stamp: "Formatted at 2026-09-25 22:40:14 on dist-test-slave-wjdr"
server_key: "27fcfd8e33369c0b2d0ab91fb2aa9a73"
server_key_iv: "a6314afbc35c3085562a67d282e47ce6"
server_key_version: "encryptionkey@0"
I20260925 22:40:14.535374 27507 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260925 22:40:14.538249 27522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:14.539666 27507 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 22:40:14.539822 27507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790375752797504-7131-0/minicluster-data/master-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790375752797504-7131-0/minicluster-data/master-0/wal
uuid: "0ec6a30510914149b2060b067c31b11e"
format_stamp: "Formatted at 2026-09-25 22:40:14 on dist-test-slave-wjdr"
server_key: "27fcfd8e33369c0b2d0ab91fb2aa9a73"
server_key_iv: "a6314afbc35c3085562a67d282e47ce6"
server_key_version: "encryptionkey@0"
I20260925 22:40:14.539952 27507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790375752797504-7131-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790375752797504-7131-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790375752797504-7131-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:40:14.572252 27507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:40:14.575724 27507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:40:14.575960 27507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:40:14.583536 27507 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44415
I20260925 22:40:14.583557 27574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44415 every 8 connection(s)
I20260925 22:40:14.584555 27507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
I20260925 22:40:14.587538 27575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:40:14.592285 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 27507
I20260925 22:40:14.592475 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790375752797504-7131-0/minicluster-data/master-0/wal/instance
I20260925 22:40:14.592820 7131 external_mini_cluster.cc:1428] Setting key 0dd6d7a4191cb621072093359880b059
I20260925 22:40:14.593827 27575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ec6a30510914149b2060b067c31b11e: Bootstrap starting.
I20260925 22:40:14.596288 27575 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ec6a30510914149b2060b067c31b11e: Neither blocks nor log segments found. Creating new log.
I20260925 22:40:14.597079 27575 log.cc:824] T 00000000000000000000000000000000 P 0ec6a30510914149b2060b067c31b11e: Log is configured to *not* fsync() on all Append() calls
I20260925 22:40:14.599426 27575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ec6a30510914149b2060b067c31b11e: No bootstrap required, opened a new log
I20260925 22:40:14.602492 27578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:14.594363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51647 (local address 127.0.0.1:44415)
0925 22:40:14.594767 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:14.594783 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:14.594817 (+ 34us) server_negotiation.cc:407] Connection header received
0925 22:40:14.595451 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:14.595467 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:14.595782 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:14.596058 (+ 276us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:14.596498 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:14.597628 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:14.598972 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:14.599455 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:14.599733 (+ 278us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:14.600748 (+ 1015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:14.600754 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:14.600773 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:40:14.601196 (+ 423us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:14.601260 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:40:14.601283 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:14.601435 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:14.601621 (+ 186us) server_negotiation.cc:299] Negotiation successful
0925 22:40:14.601759 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":133,"threads_started":1}
I20260925 22:40:14.603365 27575 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ec6a30510914149b2060b067c31b11e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ec6a30510914149b2060b067c31b11e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44415 } }
I20260925 22:40:14.603650 27575 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ec6a30510914149b2060b067c31b11e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:40:14.603700 27575 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ec6a30510914149b2060b067c31b11e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ec6a30510914149b2060b067c31b11e, State: Initialized, Role: FOLLOWER
I20260925 22:40:14.604156 27575 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ec6a30510914149b2060b067c31b11e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ec6a30510914149b2060b067c31b11e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44415 } }
I20260925 22:40:14.604337 27575 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ec6a30510914149b2060b067c31b11e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:40:14.604439 27575 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ec6a30510914149b2060b067c31b11e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:40:14.604611 27575 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ec6a30510914149b2060b067c31b11e [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:40:14.605683 7131 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:40:14.605749 27575 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ec6a30510914149b2060b067c31b11e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ec6a30510914149b2060b067c31b11e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44415 } }
I20260925 22:40:14.606092 27575 leader_election.cc:302] T 00000000000000000000000000000000 P 0ec6a30510914149b2060b067c31b11e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ec6a30510914149b2060b067c31b11e; no voters:
I20260925 22:40:14.606465 27575 leader_election.cc:288] T 00000000000000000000000000000000 P 0ec6a30510914149b2060b067c31b11e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:40:14.606578 27580 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ec6a30510914149b2060b067c31b11e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:40:14.606844 27580 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ec6a30510914149b2060b067c31b11e [term 1 LEADER]: Becoming Leader. State: Replica: 0ec6a30510914149b2060b067c31b11e, State: Running, Role: LEADER
I20260925 22:40:14.607344 27580 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ec6a30510914149b2060b067c31b11e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ec6a30510914149b2060b067c31b11e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44415 } }
I20260925 22:40:14.607798 27575 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ec6a30510914149b2060b067c31b11e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:40:14.609004 27582 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ec6a30510914149b2060b067c31b11e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ec6a30510914149b2060b067c31b11e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ec6a30510914149b2060b067c31b11e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44415 } } }
I20260925 22:40:14.609133 27582 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ec6a30510914149b2060b067c31b11e [sys.catalog]: This master's current role is: LEADER
I20260925 22:40:14.609411 27581 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ec6a30510914149b2060b067c31b11e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ec6a30510914149b2060b067c31b11e. Latest consensus state: current_term: 1 leader_uuid: "0ec6a30510914149b2060b067c31b11e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ec6a30510914149b2060b067c31b11e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44415 } } }
I20260925 22:40:14.609570 27581 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ec6a30510914149b2060b067c31b11e [sys.catalog]: This master's current role is: LEADER
I20260925 22:40:14.610809 27594 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:40:14.614015 27578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:14.608745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45793 (local address 127.0.0.1:44415)
0925 22:40:14.608853 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:14.608857 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:14.609281 (+ 424us) server_negotiation.cc:407] Connection header received
0925 22:40:14.609527 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:14.609530 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:14.609574 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:14.609624 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:14.610146 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:14.610874 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:14.612029 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:14.612374 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:14.612499 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:14.613439 (+ 940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:14.613443 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:14.613446 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:40:14.613458 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:14.613485 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:40:14.613497 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:14.613588 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:14.613826 (+ 238us) server_negotiation.cc:299] Negotiation successful
0925 22:40:14.613864 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 22:40:14.614152 27594 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:40:14.619436 27594 catalog_manager.cc:1347] Generated new cluster ID: 5b61269428674305bdbae119a406aeff
I20260925 22:40:14.619496 27594 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:40:14.625751 27594 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:40:14.626670 27594 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:40:14.637678 27594 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ec6a30510914149b2060b067c31b11e: Generated new TSK 0
I20260925 22:40:14.638501 27594 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:40:14.643723 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 27507
W20260925 22:40:14.649214 27492 connection.cc:535] client connection to 127.0.0.1:44415 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44415): Connection reset by peer (error 104)
2026-09-25T22:40:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1462 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790375752797504-7131-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:40:14 dist-test-slave-wjdr krb5kdc[27606](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:40:14 dist-test-slave-wjdr krb5kdc[27606](info): set up 2 sockets
Sep 25 22:40:14 dist-test-slave-wjdr krb5kdc[27606](info): commencing operation
krb5kdc: starting...
W20260925 22:40:15.704917 7131 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Sep 25 22:40:15 dist-test-slave-wjdr krb5kdc[27606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376015, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:40:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:40:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790375752797504-7131-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790375752797504-7131-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 22:40:15.835940 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790375752797504-7131-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790375752797504-7131-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790375752797504-7131-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35143
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790375752797504-7131-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35143
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:40:15.952409 27619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:40:15.952770 27619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:40:15.952872 27619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:40:15.956909 27619 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:40:15.956987 27619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:40:15.957034 27619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:40:15.957068 27619 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:40:15.957105 27619 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:40:15.962325 27619 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790375752797504-7131-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790375752797504-7131-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35143
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790375752797504-7131-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27619
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790375752797504-7131-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:40:15.963447 27619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:40:15.964601 27619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:40:15.973385 27627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:15.973407 27624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:15.973409 27625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:40:15.973654 27619 server_base.cc:1034] running on GCE node
I20260925 22:40:15.974298 27619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:40:15.975440 27619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:40:15.976619 27619 hybrid_clock.cc:648] HybridClock initialized: now 1790376015976607 us; error 27 us; skew 500 ppm
Sep 25 22:40:15 dist-test-slave-wjdr krb5kdc[27606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376015, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:40:15.979468 27619 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 22:40:15.980500 27619 webserver.cc:460] Webserver started at http://127.0.0.1:44081/ using document root <none> and password file <none>
I20260925 22:40:15.981120 27619 fs_manager.cc:359] Metadata directory not provided
I20260925 22:40:15.981220 27619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:40:15.981477 27619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:40:15.983640 27619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790375752797504-7131-0/minicluster-data/master-0/data/instance:
uuid: "631c3d053b5f4939b977a1afd4a58d70"
format_stamp: "Formatted at 2026-09-25 22:40:15 on dist-test-slave-wjdr"
server_key: "d913242d6ee81e1f682a1cd52e55961f"
server_key_iv: "dce7823ba05b8059fe27eac7c79f00c4"
server_key_version: "encryptionkey@0"
I20260925 22:40:15.984228 27619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790375752797504-7131-0/minicluster-data/master-0/wal/instance:
uuid: "631c3d053b5f4939b977a1afd4a58d70"
format_stamp: "Formatted at 2026-09-25 22:40:15 on dist-test-slave-wjdr"
server_key: "d913242d6ee81e1f682a1cd52e55961f"
server_key_iv: "dce7823ba05b8059fe27eac7c79f00c4"
server_key_version: "encryptionkey@0"
I20260925 22:40:15.988049 27619 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260925 22:40:15.990543 27634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:15.991766 27619 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 22:40:15.991923 27619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790375752797504-7131-0/minicluster-data/master-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790375752797504-7131-0/minicluster-data/master-0/wal
uuid: "631c3d053b5f4939b977a1afd4a58d70"
format_stamp: "Formatted at 2026-09-25 22:40:15 on dist-test-slave-wjdr"
server_key: "d913242d6ee81e1f682a1cd52e55961f"
server_key_iv: "dce7823ba05b8059fe27eac7c79f00c4"
server_key_version: "encryptionkey@0"
I20260925 22:40:15.992043 27619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790375752797504-7131-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790375752797504-7131-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790375752797504-7131-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:40:15.996665 27619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:40:15.999857 27619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:40:16.000062 27619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:40:16.007189 27619 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35143
I20260925 22:40:16.007208 27686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35143 every 8 connection(s)
I20260925 22:40:16.008111 27619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
I20260925 22:40:16.010795 27687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:40:16.012351 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 27619
I20260925 22:40:16.012516 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790375752797504-7131-0/minicluster-data/master-0/wal/instance
I20260925 22:40:16.012836 7131 external_mini_cluster.cc:1428] Setting key f3390e0744c23435420036ff047fbc35
I20260925 22:40:16.017019 27687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 631c3d053b5f4939b977a1afd4a58d70: Bootstrap starting.
I20260925 22:40:16.017935 27690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:16.014275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37079 (local address 127.0.0.1:35143)
0925 22:40:16.014725 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:16.014740 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:16.014788 (+ 48us) server_negotiation.cc:407] Connection header received
0925 22:40:16.015423 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:16.015437 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:16.015692 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:16.016014 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:16.016829 (+ 815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:16.016833 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:16.016844 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:40:16.016855 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:16.016907 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:40:16.016926 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:16.017075 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:16.017143 (+ 68us) server_negotiation.cc:299] Negotiation successful
0925 22:40:16.017287 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":129,"threads_started":1}
I20260925 22:40:16.019529 27687 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 631c3d053b5f4939b977a1afd4a58d70: Neither blocks nor log segments found. Creating new log.
I20260925 22:40:16.020342 27687 log.cc:824] T 00000000000000000000000000000000 P 631c3d053b5f4939b977a1afd4a58d70: Log is configured to *not* fsync() on all Append() calls
I20260925 22:40:16.020745 7131 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:40:16.022492 27687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 631c3d053b5f4939b977a1afd4a58d70: No bootstrap required, opened a new log
I20260925 22:40:16.024545 27690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:16.022798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45507 (local address 127.0.0.1:35143)
0925 22:40:16.022906 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:16.022910 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:16.023154 (+ 244us) server_negotiation.cc:407] Connection header received
0925 22:40:16.023289 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:16.023291 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:16.023321 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:16.023379 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:16.024091 (+ 712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:16.024094 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:16.024096 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:40:16.024110 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:16.024133 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:40:16.024143 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:16.024208 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:16.024330 (+ 122us) server_negotiation.cc:299] Negotiation successful
0925 22:40:16.024361 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 22:40:16.026909 27687 raft_consensus.cc:348] T 00000000000000000000000000000000 P 631c3d053b5f4939b977a1afd4a58d70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "631c3d053b5f4939b977a1afd4a58d70" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35143 } }
I20260925 22:40:16.027134 27687 raft_consensus.cc:374] T 00000000000000000000000000000000 P 631c3d053b5f4939b977a1afd4a58d70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:40:16.027206 27687 raft_consensus.cc:729] T 00000000000000000000000000000000 P 631c3d053b5f4939b977a1afd4a58d70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 631c3d053b5f4939b977a1afd4a58d70, State: Initialized, Role: FOLLOWER
I20260925 22:40:16.027668 27687 consensus_queue.cc:260] T 00000000000000000000000000000000 P 631c3d053b5f4939b977a1afd4a58d70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "631c3d053b5f4939b977a1afd4a58d70" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35143 } }
I20260925 22:40:16.027805 27687 raft_consensus.cc:388] T 00000000000000000000000000000000 P 631c3d053b5f4939b977a1afd4a58d70 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:40:16.027878 27687 raft_consensus.cc:482] T 00000000000000000000000000000000 P 631c3d053b5f4939b977a1afd4a58d70 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:40:16.028004 27687 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 631c3d053b5f4939b977a1afd4a58d70 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:40:16.029057 27687 raft_consensus.cc:504] T 00000000000000000000000000000000 P 631c3d053b5f4939b977a1afd4a58d70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "631c3d053b5f4939b977a1afd4a58d70" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35143 } }
I20260925 22:40:16.029418 27687 leader_election.cc:302] T 00000000000000000000000000000000 P 631c3d053b5f4939b977a1afd4a58d70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 631c3d053b5f4939b977a1afd4a58d70; no voters:
I20260925 22:40:16.029739 27687 leader_election.cc:288] T 00000000000000000000000000000000 P 631c3d053b5f4939b977a1afd4a58d70 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:40:16.029925 27697 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 631c3d053b5f4939b977a1afd4a58d70 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:40:16.030189 27697 raft_consensus.cc:686] T 00000000000000000000000000000000 P 631c3d053b5f4939b977a1afd4a58d70 [term 1 LEADER]: Becoming Leader. State: Replica: 631c3d053b5f4939b977a1afd4a58d70, State: Running, Role: LEADER
I20260925 22:40:16.030581 27697 consensus_queue.cc:237] T 00000000000000000000000000000000 P 631c3d053b5f4939b977a1afd4a58d70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "631c3d053b5f4939b977a1afd4a58d70" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35143 } }
I20260925 22:40:16.031270 27687 sys_catalog.cc:564] T 00000000000000000000000000000000 P 631c3d053b5f4939b977a1afd4a58d70 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:40:16.032625 27699 sys_catalog.cc:455] T 00000000000000000000000000000000 P 631c3d053b5f4939b977a1afd4a58d70 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 631c3d053b5f4939b977a1afd4a58d70. Latest consensus state: current_term: 1 leader_uuid: "631c3d053b5f4939b977a1afd4a58d70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "631c3d053b5f4939b977a1afd4a58d70" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35143 } } }
I20260925 22:40:16.032807 27699 sys_catalog.cc:458] T 00000000000000000000000000000000 P 631c3d053b5f4939b977a1afd4a58d70 [sys.catalog]: This master's current role is: LEADER
I20260925 22:40:16.033233 27706 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:40:16.033169 27698 sys_catalog.cc:455] T 00000000000000000000000000000000 P 631c3d053b5f4939b977a1afd4a58d70 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "631c3d053b5f4939b977a1afd4a58d70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "631c3d053b5f4939b977a1afd4a58d70" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35143 } } }
I20260925 22:40:16.034046 27698 sys_catalog.cc:458] T 00000000000000000000000000000000 P 631c3d053b5f4939b977a1afd4a58d70 [sys.catalog]: This master's current role is: LEADER
I20260925 22:40:16.036026 27706 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:40:16.041188 27706 catalog_manager.cc:1347] Generated new cluster ID: f6ae29189056445a9ea27ba40a4529e7
I20260925 22:40:16.041278 27706 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:40:16.063875 27706 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:40:16.064206 27706 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:40:16.077318 27706 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 631c3d053b5f4939b977a1afd4a58d70: Generated new TSK 0
I20260925 22:40:16.078150 27706 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:40:16.081534 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 27619
2026-09-25T22:40:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1437 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790375752797504-7131-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:40:16 dist-test-slave-wjdr krb5kdc[27718](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:40:16 dist-test-slave-wjdr krb5kdc[27718](info): set up 2 sockets
Sep 25 22:40:16 dist-test-slave-wjdr krb5kdc[27718](info): commencing operation
krb5kdc: starting...
W20260925 22:40:17.169122 7131 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Sep 25 22:40:17 dist-test-slave-wjdr krb5kdc[27718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:40:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:40:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790375752797504-7131-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790375752797504-7131-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 22:40:17.304452 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790375752797504-7131-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790375752797504-7131-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790375752797504-7131-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34897
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790375752797504-7131-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34897
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:40:17.431608 27731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:40:17.431933 27731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:40:17.431978 27731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:40:17.436152 27731 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:40:17.436239 27731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:40:17.436259 27731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:40:17.436275 27731 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:40:17.436291 27731 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:40:17.442162 27731 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790375752797504-7131-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790375752797504-7131-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34897
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790375752797504-7131-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790375752797504-7131-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:40:17.443522 27731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:40:17.444777 27731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:40:17.455087 27737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:17.455153 27736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:17.455093 27739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:40:17.455502 27731 server_base.cc:1034] running on GCE node
I20260925 22:40:17.456270 27731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:40:17.457669 27731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:40:17.458877 27731 hybrid_clock.cc:648] HybridClock initialized: now 1790376017458862 us; error 35 us; skew 500 ppm
Sep 25 22:40:17 dist-test-slave-wjdr krb5kdc[27718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376017, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:40:17.462110 27731 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 22:40:17.463171 27731 webserver.cc:460] Webserver started at http://127.0.0.1:43387/ using document root <none> and password file <none>
I20260925 22:40:17.463851 27731 fs_manager.cc:359] Metadata directory not provided
I20260925 22:40:17.463949 27731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:40:17.464215 27731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:40:17.466660 27731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790375752797504-7131-0/minicluster-data/master-0/data/instance:
uuid: "1058bb055140421f97a2c68482316293"
format_stamp: "Formatted at 2026-09-25 22:40:17 on dist-test-slave-wjdr"
server_key: "1f3fdbd7e175fe9c65ef660a820a1739"
server_key_iv: "6536e7ad304170c476289074aa226b8c"
server_key_version: "encryptionkey@0"
I20260925 22:40:17.467304 27731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790375752797504-7131-0/minicluster-data/master-0/wal/instance:
uuid: "1058bb055140421f97a2c68482316293"
format_stamp: "Formatted at 2026-09-25 22:40:17 on dist-test-slave-wjdr"
server_key: "1f3fdbd7e175fe9c65ef660a820a1739"
server_key_iv: "6536e7ad304170c476289074aa226b8c"
server_key_version: "encryptionkey@0"
I20260925 22:40:17.471817 27731 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 22:40:17.474713 27746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:17.476181 27731 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:40:17.476338 27731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790375752797504-7131-0/minicluster-data/master-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790375752797504-7131-0/minicluster-data/master-0/wal
uuid: "1058bb055140421f97a2c68482316293"
format_stamp: "Formatted at 2026-09-25 22:40:17 on dist-test-slave-wjdr"
server_key: "1f3fdbd7e175fe9c65ef660a820a1739"
server_key_iv: "6536e7ad304170c476289074aa226b8c"
server_key_version: "encryptionkey@0"
I20260925 22:40:17.476502 27731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790375752797504-7131-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790375752797504-7131-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790375752797504-7131-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:40:17.489146 27731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:40:17.493793 27731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:40:17.494096 27731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:40:17.502996 27731 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34897
I20260925 22:40:17.503014 27798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34897 every 8 connection(s)
I20260925 22:40:17.504142 27731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
I20260925 22:40:17.507570 27799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:40:17.512321 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 27731
I20260925 22:40:17.512502 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790375752797504-7131-0/minicluster-data/master-0/wal/instance
I20260925 22:40:17.512861 7131 external_mini_cluster.cc:1428] Setting key 3515f1fdcb5fd4b64fc54c20a8203d13
I20260925 22:40:17.515053 27799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1058bb055140421f97a2c68482316293: Bootstrap starting.
I20260925 22:40:17.517879 27799 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1058bb055140421f97a2c68482316293: Neither blocks nor log segments found. Creating new log.
I20260925 22:40:17.518848 27799 log.cc:824] T 00000000000000000000000000000000 P 1058bb055140421f97a2c68482316293: Log is configured to *not* fsync() on all Append() calls
I20260925 22:40:17.521065 27799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1058bb055140421f97a2c68482316293: No bootstrap required, opened a new log
Sep 25 22:40:17 dist-test-slave-wjdr krb5kdc[27718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 22:40:17.525027 27799 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1058bb055140421f97a2c68482316293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1058bb055140421f97a2c68482316293" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34897 } }
I20260925 22:40:17.525277 27799 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1058bb055140421f97a2c68482316293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:40:17.525346 27799 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1058bb055140421f97a2c68482316293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1058bb055140421f97a2c68482316293, State: Initialized, Role: FOLLOWER
I20260925 22:40:17.525909 27799 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1058bb055140421f97a2c68482316293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1058bb055140421f97a2c68482316293" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34897 } }
I20260925 22:40:17.526086 27799 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1058bb055140421f97a2c68482316293 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:40:17.526188 27799 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1058bb055140421f97a2c68482316293 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:40:17.526322 27799 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1058bb055140421f97a2c68482316293 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:40:17.527766 27799 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1058bb055140421f97a2c68482316293 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1058bb055140421f97a2c68482316293" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34897 } }
I20260925 22:40:17.528201 27799 leader_election.cc:302] T 00000000000000000000000000000000 P 1058bb055140421f97a2c68482316293 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1058bb055140421f97a2c68482316293; no voters:
I20260925 22:40:17.528636 27799 leader_election.cc:288] T 00000000000000000000000000000000 P 1058bb055140421f97a2c68482316293 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:40:17.528739 27804 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1058bb055140421f97a2c68482316293 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:40:17.529018 27804 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1058bb055140421f97a2c68482316293 [term 1 LEADER]: Becoming Leader. State: Replica: 1058bb055140421f97a2c68482316293, State: Running, Role: LEADER
I20260925 22:40:17.529439 27804 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1058bb055140421f97a2c68482316293 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1058bb055140421f97a2c68482316293" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34897 } }
I20260925 22:40:17.530237 27799 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1058bb055140421f97a2c68482316293 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:40:17.530314 27802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:17.514678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55711 (local address 127.0.0.1:34897)
0925 22:40:17.515253 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:17.515272 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:17.515314 (+ 42us) server_negotiation.cc:407] Connection header received
0925 22:40:17.516030 (+ 716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:17.516048 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:17.516439 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:17.516899 (+ 460us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:17.517810 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:17.519063 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:17.519986 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:17.520566 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:17.520740 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:17.523760 (+ 3020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:17.523765 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:17.523779 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:17.523807 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:17.526578 (+ 2771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:17.527097 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:17.527102 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:17.527107 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:17.527241 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:17.527587 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:17.527589 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:17.527590 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:17.528103 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:17.528256 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:17.528736 (+ 480us) server_negotiation.cc:299] Negotiation successful
0925 22:40:17.529159 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":165,"threads_started":1}
I20260925 22:40:17.531442 27805 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1058bb055140421f97a2c68482316293 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1058bb055140421f97a2c68482316293" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1058bb055140421f97a2c68482316293" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34897 } } }
I20260925 22:40:17.531816 27805 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1058bb055140421f97a2c68482316293 [sys.catalog]: This master's current role is: LEADER
I20260925 22:40:17.531718 27806 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1058bb055140421f97a2c68482316293 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1058bb055140421f97a2c68482316293. Latest consensus state: current_term: 1 leader_uuid: "1058bb055140421f97a2c68482316293" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1058bb055140421f97a2c68482316293" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34897 } } }
I20260925 22:40:17.532091 27806 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1058bb055140421f97a2c68482316293 [sys.catalog]: This master's current role is: LEADER
I20260925 22:40:17.532641 27813 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:40:17.535070 7131 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:40:17.537026 27813 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:40:17.543821 27813 catalog_manager.cc:1347] Generated new cluster ID: 45914a078e0c4e7daa21ffd59094dc38
I20260925 22:40:17.543900 27813 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:40:17.548534 27802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:17.539270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53471 (local address 127.0.0.1:34897)
0925 22:40:17.539408 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:17.539415 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:17.539814 (+ 399us) server_negotiation.cc:407] Connection header received
0925 22:40:17.539990 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:17.539994 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:17.540056 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:17.540151 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:17.540986 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:17.541804 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:17.542680 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:17.542941 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:17.543061 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:17.544924 (+ 1863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:17.544930 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:17.544936 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:17.544965 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:17.546709 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:17.547191 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:17.547195 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:17.547197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:17.547290 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:17.547571 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:17.547574 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:17.547576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:17.547811 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:17.547954 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:17.548186 (+ 232us) server_negotiation.cc:299] Negotiation successful
0925 22:40:17.548322 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 22:40:17.564738 27813 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:40:17.566294 27813 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:40:17.578018 27813 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1058bb055140421f97a2c68482316293: Generated new TSK 0
I20260925 22:40:17.579118 27813 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:40:17.586133 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 27731
2026-09-25T22:40:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1505 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790375752797504-7131-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:40:17 dist-test-slave-wjdr krb5kdc[27830](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:40:17 dist-test-slave-wjdr krb5kdc[27830](info): set up 2 sockets
Sep 25 22:40:17 dist-test-slave-wjdr krb5kdc[27830](info): commencing operation
krb5kdc: starting...
W20260925 22:40:18.629639 7131 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Sep 25 22:40:18 dist-test-slave-wjdr krb5kdc[27830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:40:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:40:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790375752797504-7131-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790375752797504-7131-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 22:40:18.772573 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790375752797504-7131-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790375752797504-7131-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790375752797504-7131-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33269
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790375752797504-7131-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33269
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:40:18.906404 27843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:40:18.906723 27843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:40:18.906769 27843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:40:18.910940 27843 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:40:18.911011 27843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:40:18.911031 27843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:40:18.911047 27843 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:40:18.911062 27843 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:40:18.917073 27843 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790375752797504-7131-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790375752797504-7131-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33269
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790375752797504-7131-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27843
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790375752797504-7131-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:40:18.918408 27843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:40:18.919636 27843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:40:18.929301 27848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:18.929301 27851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:18.929368 27849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:40:18.929752 27843 server_base.cc:1034] running on GCE node
I20260925 22:40:18.930375 27843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:40:18.931742 27843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:40:18.932952 27843 hybrid_clock.cc:648] HybridClock initialized: now 1790376018932926 us; error 45 us; skew 500 ppm
Sep 25 22:40:18 dist-test-slave-wjdr krb5kdc[27830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376018, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:40:18.936250 27843 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 22:40:18.937458 27843 webserver.cc:460] Webserver started at http://127.0.0.1:45475/ using document root <none> and password file <none>
I20260925 22:40:18.938158 27843 fs_manager.cc:359] Metadata directory not provided
I20260925 22:40:18.938263 27843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:40:18.938584 27843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:40:18.941156 27843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790375752797504-7131-0/minicluster-data/master-0/data/instance:
uuid: "497ea110169e472cac226695c8c73a34"
format_stamp: "Formatted at 2026-09-25 22:40:18 on dist-test-slave-wjdr"
server_key: "15ff0beed76e418fe3b4b8a99611ca8f"
server_key_iv: "1ae2d5f36ed9f444048fc63c47df2381"
server_key_version: "encryptionkey@0"
I20260925 22:40:18.941794 27843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790375752797504-7131-0/minicluster-data/master-0/wal/instance:
uuid: "497ea110169e472cac226695c8c73a34"
format_stamp: "Formatted at 2026-09-25 22:40:18 on dist-test-slave-wjdr"
server_key: "15ff0beed76e418fe3b4b8a99611ca8f"
server_key_iv: "1ae2d5f36ed9f444048fc63c47df2381"
server_key_version: "encryptionkey@0"
I20260925 22:40:18.946267 27843 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:40:18.949141 27858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:18.950770 27843 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:40:18.950991 27843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790375752797504-7131-0/minicluster-data/master-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790375752797504-7131-0/minicluster-data/master-0/wal
uuid: "497ea110169e472cac226695c8c73a34"
format_stamp: "Formatted at 2026-09-25 22:40:18 on dist-test-slave-wjdr"
server_key: "15ff0beed76e418fe3b4b8a99611ca8f"
server_key_iv: "1ae2d5f36ed9f444048fc63c47df2381"
server_key_version: "encryptionkey@0"
I20260925 22:40:18.951157 27843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790375752797504-7131-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790375752797504-7131-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790375752797504-7131-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:40:18.970269 27843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:40:18.974555 27843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:40:18.974824 27843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:40:18.984560 27843 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33269
I20260925 22:40:18.984556 27910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33269 every 8 connection(s)
I20260925 22:40:18.985855 27843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
I20260925 22:40:18.989468 27911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:40:18.993134 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 27843
I20260925 22:40:18.993304 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790375752797504-7131-0/minicluster-data/master-0/wal/instance
I20260925 22:40:18.993652 7131 external_mini_cluster.cc:1428] Setting key 3fd521c4fd446ba5c99e9283bc3be0a5
I20260925 22:40:18.996968 27911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 497ea110169e472cac226695c8c73a34: Bootstrap starting.
I20260925 22:40:19.000263 27911 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 497ea110169e472cac226695c8c73a34: Neither blocks nor log segments found. Creating new log.
I20260925 22:40:19.001338 27911 log.cc:824] T 00000000000000000000000000000000 P 497ea110169e472cac226695c8c73a34: Log is configured to *not* fsync() on all Append() calls
I20260925 22:40:19.004050 27911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 497ea110169e472cac226695c8c73a34: No bootstrap required, opened a new log
I20260925 22:40:19.004436 27914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:18.995297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34995 (local address 127.0.0.1:33269)
0925 22:40:18.995778 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:18.995797 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:18.995838 (+ 41us) server_negotiation.cc:407] Connection header received
0925 22:40:18.996659 (+ 821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:18.996681 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:18.997142 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:18.997487 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:18.997868 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:18.998918 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:19.000227 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:19.000914 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:19.001049 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:19.002133 (+ 1084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:19.002140 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:19.002165 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:40:19.002741 (+ 576us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:19.002821 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:40:19.002862 (+ 41us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:19.002953 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:19.003145 (+ 192us) server_negotiation.cc:299] Negotiation successful
0925 22:40:19.003434 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":161,"threads_started":1}
I20260925 22:40:19.008239 7131 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:40:19.009145 27911 raft_consensus.cc:348] T 00000000000000000000000000000000 P 497ea110169e472cac226695c8c73a34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "497ea110169e472cac226695c8c73a34" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33269 } }
I20260925 22:40:19.009413 27911 raft_consensus.cc:374] T 00000000000000000000000000000000 P 497ea110169e472cac226695c8c73a34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:40:19.009488 27911 raft_consensus.cc:729] T 00000000000000000000000000000000 P 497ea110169e472cac226695c8c73a34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 497ea110169e472cac226695c8c73a34, State: Initialized, Role: FOLLOWER
I20260925 22:40:19.010044 27911 consensus_queue.cc:260] T 00000000000000000000000000000000 P 497ea110169e472cac226695c8c73a34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "497ea110169e472cac226695c8c73a34" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33269 } }
I20260925 22:40:19.010198 27911 raft_consensus.cc:388] T 00000000000000000000000000000000 P 497ea110169e472cac226695c8c73a34 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:40:19.010298 27911 raft_consensus.cc:482] T 00000000000000000000000000000000 P 497ea110169e472cac226695c8c73a34 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:40:19.010430 27911 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 497ea110169e472cac226695c8c73a34 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:40:19.011720 27911 raft_consensus.cc:504] T 00000000000000000000000000000000 P 497ea110169e472cac226695c8c73a34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "497ea110169e472cac226695c8c73a34" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33269 } }
I20260925 22:40:19.012151 27911 leader_election.cc:302] T 00000000000000000000000000000000 P 497ea110169e472cac226695c8c73a34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 497ea110169e472cac226695c8c73a34; no voters:
I20260925 22:40:19.012581 27911 leader_election.cc:288] T 00000000000000000000000000000000 P 497ea110169e472cac226695c8c73a34 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:40:19.012687 27921 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 497ea110169e472cac226695c8c73a34 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:40:19.012871 27921 raft_consensus.cc:686] T 00000000000000000000000000000000 P 497ea110169e472cac226695c8c73a34 [term 1 LEADER]: Becoming Leader. State: Replica: 497ea110169e472cac226695c8c73a34, State: Running, Role: LEADER
I20260925 22:40:19.013260 27921 consensus_queue.cc:237] T 00000000000000000000000000000000 P 497ea110169e472cac226695c8c73a34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "497ea110169e472cac226695c8c73a34" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33269 } }
I20260925 22:40:19.014189 27911 sys_catalog.cc:564] T 00000000000000000000000000000000 P 497ea110169e472cac226695c8c73a34 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:40:19.014701 27923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 497ea110169e472cac226695c8c73a34 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 497ea110169e472cac226695c8c73a34. Latest consensus state: current_term: 1 leader_uuid: "497ea110169e472cac226695c8c73a34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "497ea110169e472cac226695c8c73a34" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33269 } } }
I20260925 22:40:19.014870 27923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 497ea110169e472cac226695c8c73a34 [sys.catalog]: This master's current role is: LEADER
I20260925 22:40:19.015744 27922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 497ea110169e472cac226695c8c73a34 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "497ea110169e472cac226695c8c73a34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "497ea110169e472cac226695c8c73a34" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33269 } } }
I20260925 22:40:19.015910 27922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 497ea110169e472cac226695c8c73a34 [sys.catalog]: This master's current role is: LEADER
I20260925 22:40:19.016078 27914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:19.010895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39207 (local address 127.0.0.1:33269)
0925 22:40:19.011031 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:19.011036 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:19.011275 (+ 239us) server_negotiation.cc:407] Connection header received
0925 22:40:19.011444 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:19.011447 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:19.011508 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:19.011600 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:19.011917 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:19.012608 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:19.013788 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:19.014139 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:19.014234 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:19.015201 (+ 967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:19.015207 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:19.015213 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:40:19.015227 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:19.015264 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:40:19.015280 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:19.015352 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:19.015816 (+ 464us) server_negotiation.cc:299] Negotiation successful
0925 22:40:19.015873 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 22:40:19.016273 27925 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:40:19.019652 27925 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:40:19.026095 27925 catalog_manager.cc:1347] Generated new cluster ID: 44e2452497be46528765f02fb7abafd9
I20260925 22:40:19.026180 27925 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:40:19.040690 27925 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:40:19.042241 27925 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:40:19.060741 27925 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 497ea110169e472cac226695c8c73a34: Generated new TSK 0
I20260925 22:40:19.061731 27925 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:40:19.074011 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 27843
2026-09-25T22:40:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1488 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790375752797504-7131-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:40:19 dist-test-slave-wjdr krb5kdc[27942](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:40:19 dist-test-slave-wjdr krb5kdc[27942](info): set up 2 sockets
Sep 25 22:40:19 dist-test-slave-wjdr krb5kdc[27942](info): commencing operation
krb5kdc: starting...
W20260925 22:40:20.149897 7131 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Sep 25 22:40:20 dist-test-slave-wjdr krb5kdc[27942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:40:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:40:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790375752797504-7131-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790375752797504-7131-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 22:40:20.284163 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790375752797504-7131-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790375752797504-7131-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790375752797504-7131-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44299
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790375752797504-7131-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44299
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:40:20.410364 27955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:40:20.410669 27955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:40:20.410713 27955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:40:20.414909 27955 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:40:20.414988 27955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:40:20.415007 27955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:40:20.415025 27955 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:40:20.415091 27955 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:40:20.420636 27955 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790375752797504-7131-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790375752797504-7131-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44299
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790375752797504-7131-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27955
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790375752797504-7131-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:40:20.422003 27955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:40:20.423274 27955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:40:20.433259 27961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:20.433300 27960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:20.433279 27963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:40:20.433535 27955 server_base.cc:1034] running on GCE node
I20260925 22:40:20.434278 27955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:40:20.435567 27955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:40:20.436760 27955 hybrid_clock.cc:648] HybridClock initialized: now 1790376020436732 us; error 46 us; skew 500 ppm
Sep 25 22:40:20 dist-test-slave-wjdr krb5kdc[27942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376020, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:40:20.446694 27955 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 22:40:20.447854 27955 webserver.cc:460] Webserver started at http://127.0.0.1:41693/ using document root <none> and password file <none>
I20260925 22:40:20.448611 27955 fs_manager.cc:359] Metadata directory not provided
I20260925 22:40:20.448714 27955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:40:20.448973 27955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:40:20.451434 27955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790375752797504-7131-0/minicluster-data/master-0/data/instance:
uuid: "4dfcc847f2834250a535aa93bea49573"
format_stamp: "Formatted at 2026-09-25 22:40:20 on dist-test-slave-wjdr"
server_key: "1e8ac564329ae76ce7c5f9c9833695b1"
server_key_iv: "43124f5099e7279a3c6f471b358904a9"
server_key_version: "encryptionkey@0"
I20260925 22:40:20.452095 27955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790375752797504-7131-0/minicluster-data/master-0/wal/instance:
uuid: "4dfcc847f2834250a535aa93bea49573"
format_stamp: "Formatted at 2026-09-25 22:40:20 on dist-test-slave-wjdr"
server_key: "1e8ac564329ae76ce7c5f9c9833695b1"
server_key_iv: "43124f5099e7279a3c6f471b358904a9"
server_key_version: "encryptionkey@0"
I20260925 22:40:20.456578 27955 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260925 22:40:20.459462 27970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:20.460938 27955 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:40:20.461093 27955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790375752797504-7131-0/minicluster-data/master-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790375752797504-7131-0/minicluster-data/master-0/wal
uuid: "4dfcc847f2834250a535aa93bea49573"
format_stamp: "Formatted at 2026-09-25 22:40:20 on dist-test-slave-wjdr"
server_key: "1e8ac564329ae76ce7c5f9c9833695b1"
server_key_iv: "43124f5099e7279a3c6f471b358904a9"
server_key_version: "encryptionkey@0"
I20260925 22:40:20.461225 27955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790375752797504-7131-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790375752797504-7131-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790375752797504-7131-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:40:20.466418 27955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:40:20.469877 27955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:40:20.470081 27955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:40:20.478201 27955 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44299
I20260925 22:40:20.478235 28022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44299 every 8 connection(s)
I20260925 22:40:20.479313 27955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
I20260925 22:40:20.482054 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 27955
I20260925 22:40:20.482227 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790375752797504-7131-0/minicluster-data/master-0/wal/instance
I20260925 22:40:20.482571 7131 external_mini_cluster.cc:1428] Setting key 34a0ef4e18b0cd46cdefd3e3a91cbf9b
I20260925 22:40:20.482939 28023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:40:20.488363 28026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:20.484217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51253 (local address 127.0.0.1:44299)
0925 22:40:20.484700 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:20.484716 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:20.484758 (+ 42us) server_negotiation.cc:407] Connection header received
0925 22:40:20.485495 (+ 737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:20.485512 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:20.485829 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:20.486197 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:20.487064 (+ 867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:20.487068 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:20.487080 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:40:20.487091 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:20.487153 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:40:20.487174 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:20.487333 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:20.487409 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 22:40:20.487579 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":176,"threads_started":1}
I20260925 22:40:20.490387 28023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4dfcc847f2834250a535aa93bea49573: Bootstrap starting.
I20260925 22:40:20.491353 7131 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:40:20.493268 28023 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4dfcc847f2834250a535aa93bea49573: Neither blocks nor log segments found. Creating new log.
I20260925 22:40:20.494261 28023 log.cc:824] T 00000000000000000000000000000000 P 4dfcc847f2834250a535aa93bea49573: Log is configured to *not* fsync() on all Append() calls
I20260925 22:40:20.495509 28026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:20.493754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57855 (local address 127.0.0.1:44299)
0925 22:40:20.493888 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:20.493893 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:20.494114 (+ 221us) server_negotiation.cc:407] Connection header received
0925 22:40:20.494242 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:20.494244 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:20.494273 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:20.494333 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:20.495084 (+ 751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:20.495087 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:20.495089 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:40:20.495099 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:20.495123 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:40:20.495136 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:20.495240 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:20.495316 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 22:40:20.495358 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 22:40:20.496695 28023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4dfcc847f2834250a535aa93bea49573: No bootstrap required, opened a new log
I20260925 22:40:20.500525 28023 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4dfcc847f2834250a535aa93bea49573 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dfcc847f2834250a535aa93bea49573" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44299 } }
I20260925 22:40:20.500790 28023 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4dfcc847f2834250a535aa93bea49573 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:40:20.500849 28023 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4dfcc847f2834250a535aa93bea49573 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dfcc847f2834250a535aa93bea49573, State: Initialized, Role: FOLLOWER
I20260925 22:40:20.501396 28023 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4dfcc847f2834250a535aa93bea49573 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dfcc847f2834250a535aa93bea49573" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44299 } }
I20260925 22:40:20.501569 28023 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4dfcc847f2834250a535aa93bea49573 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:40:20.501629 28023 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4dfcc847f2834250a535aa93bea49573 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:40:20.501708 28023 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4dfcc847f2834250a535aa93bea49573 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:40:20.502611 28023 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4dfcc847f2834250a535aa93bea49573 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dfcc847f2834250a535aa93bea49573" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44299 } }
I20260925 22:40:20.502926 28023 leader_election.cc:302] T 00000000000000000000000000000000 P 4dfcc847f2834250a535aa93bea49573 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4dfcc847f2834250a535aa93bea49573; no voters:
I20260925 22:40:20.503196 28023 leader_election.cc:288] T 00000000000000000000000000000000 P 4dfcc847f2834250a535aa93bea49573 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:40:20.503311 28033 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4dfcc847f2834250a535aa93bea49573 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:40:20.503552 28033 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4dfcc847f2834250a535aa93bea49573 [term 1 LEADER]: Becoming Leader. State: Replica: 4dfcc847f2834250a535aa93bea49573, State: Running, Role: LEADER
I20260925 22:40:20.503911 28033 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4dfcc847f2834250a535aa93bea49573 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dfcc847f2834250a535aa93bea49573" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44299 } }
I20260925 22:40:20.504640 28023 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4dfcc847f2834250a535aa93bea49573 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:40:20.505848 28035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4dfcc847f2834250a535aa93bea49573 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4dfcc847f2834250a535aa93bea49573. Latest consensus state: current_term: 1 leader_uuid: "4dfcc847f2834250a535aa93bea49573" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dfcc847f2834250a535aa93bea49573" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44299 } } }
I20260925 22:40:20.505987 28035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4dfcc847f2834250a535aa93bea49573 [sys.catalog]: This master's current role is: LEADER
I20260925 22:40:20.506384 28042 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:40:20.506348 28034 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4dfcc847f2834250a535aa93bea49573 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4dfcc847f2834250a535aa93bea49573" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dfcc847f2834250a535aa93bea49573" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44299 } } }
I20260925 22:40:20.507090 28034 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4dfcc847f2834250a535aa93bea49573 [sys.catalog]: This master's current role is: LEADER
I20260925 22:40:20.510011 28042 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:40:20.516382 28042 catalog_manager.cc:1347] Generated new cluster ID: 2f555292059140fd9fb3b8f1ece361ba
I20260925 22:40:20.516489 28042 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:40:20.534721 28042 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:40:20.535167 28042 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:40:20.548024 28042 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4dfcc847f2834250a535aa93bea49573: Generated new TSK 0
I20260925 22:40:20.548985 28042 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:40:20.556989 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 27955
2026-09-25T22:40:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1482 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790375752797504-7131-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:40:20 dist-test-slave-wjdr krb5kdc[28054](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:40:20 dist-test-slave-wjdr krb5kdc[28054](info): set up 2 sockets
Sep 25 22:40:20 dist-test-slave-wjdr krb5kdc[28054](info): commencing operation
krb5kdc: starting...
W20260925 22:40:21.625304 7131 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Sep 25 22:40:21 dist-test-slave-wjdr krb5kdc[28054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:40:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:40:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790375752797504-7131-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790375752797504-7131-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 22:40:21.757370 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790375752797504-7131-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790375752797504-7131-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790375752797504-7131-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38861
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790375752797504-7131-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38861
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:40:21.886610 28067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:40:21.886957 28067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:40:21.887003 28067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:40:21.891232 28067 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:40:21.891304 28067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:40:21.891324 28067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:40:21.891341 28067 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:40:21.891357 28067 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:40:21.897035 28067 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790375752797504-7131-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790375752797504-7131-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38861
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790375752797504-7131-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28067
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790375752797504-7131-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:40:21.898293 28067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:40:21.899478 28067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:40:21.909206 28073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:21.909247 28072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:21.909231 28075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:40:21.909365 28067 server_base.cc:1034] running on GCE node
I20260925 22:40:21.910288 28067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:40:21.911607 28067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:40:21.912768 28067 hybrid_clock.cc:648] HybridClock initialized: now 1790376021912739 us; error 47 us; skew 500 ppm
Sep 25 22:40:21 dist-test-slave-wjdr krb5kdc[28054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376021, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:40:21.915845 28067 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 22:40:21.917007 28067 webserver.cc:460] Webserver started at http://127.0.0.1:34643/ using document root <none> and password file <none>
I20260925 22:40:21.917688 28067 fs_manager.cc:359] Metadata directory not provided
I20260925 22:40:21.917780 28067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:40:21.918056 28067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:40:21.920899 28067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790375752797504-7131-0/minicluster-data/master-0/data/instance:
uuid: "313b1c09ba64457cac1e4ebda0920d1f"
format_stamp: "Formatted at 2026-09-25 22:40:21 on dist-test-slave-wjdr"
server_key: "3eba90ae39eadeb333027430807c87cc"
server_key_iv: "95df883f47b67d9735234556c5389065"
server_key_version: "encryptionkey@0"
I20260925 22:40:21.921561 28067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790375752797504-7131-0/minicluster-data/master-0/wal/instance:
uuid: "313b1c09ba64457cac1e4ebda0920d1f"
format_stamp: "Formatted at 2026-09-25 22:40:21 on dist-test-slave-wjdr"
server_key: "3eba90ae39eadeb333027430807c87cc"
server_key_iv: "95df883f47b67d9735234556c5389065"
server_key_version: "encryptionkey@0"
I20260925 22:40:21.926218 28067 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:40:21.929219 28082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:21.930719 28067 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:40:21.930891 28067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790375752797504-7131-0/minicluster-data/master-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790375752797504-7131-0/minicluster-data/master-0/wal
uuid: "313b1c09ba64457cac1e4ebda0920d1f"
format_stamp: "Formatted at 2026-09-25 22:40:21 on dist-test-slave-wjdr"
server_key: "3eba90ae39eadeb333027430807c87cc"
server_key_iv: "95df883f47b67d9735234556c5389065"
server_key_version: "encryptionkey@0"
I20260925 22:40:21.931037 28067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790375752797504-7131-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790375752797504-7131-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790375752797504-7131-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:40:21.952128 28067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:40:21.956346 28067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:40:21.956645 28067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:40:21.965235 28067 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38861
I20260925 22:40:21.965312 28134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38861 every 8 connection(s)
I20260925 22:40:21.966427 28067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
I20260925 22:40:21.969798 28135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:40:21.975654 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 28067
I20260925 22:40:21.975832 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790375752797504-7131-0/minicluster-data/master-0/wal/instance
I20260925 22:40:21.976181 7131 external_mini_cluster.cc:1428] Setting key 1490ba8413c0f49919285e1aaa56ade6
I20260925 22:40:21.976769 28135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 313b1c09ba64457cac1e4ebda0920d1f: Bootstrap starting.
I20260925 22:40:21.979833 28135 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 313b1c09ba64457cac1e4ebda0920d1f: Neither blocks nor log segments found. Creating new log.
I20260925 22:40:21.980786 28135 log.cc:824] T 00000000000000000000000000000000 P 313b1c09ba64457cac1e4ebda0920d1f: Log is configured to *not* fsync() on all Append() calls
I20260925 22:40:21.982959 28135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 313b1c09ba64457cac1e4ebda0920d1f: No bootstrap required, opened a new log
Sep 25 22:40:21 dist-test-slave-wjdr krb5kdc[28054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 22:40:21.987895 28135 raft_consensus.cc:348] T 00000000000000000000000000000000 P 313b1c09ba64457cac1e4ebda0920d1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "313b1c09ba64457cac1e4ebda0920d1f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38861 } }
I20260925 22:40:21.988273 28135 raft_consensus.cc:374] T 00000000000000000000000000000000 P 313b1c09ba64457cac1e4ebda0920d1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:40:21.988348 28135 raft_consensus.cc:729] T 00000000000000000000000000000000 P 313b1c09ba64457cac1e4ebda0920d1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 313b1c09ba64457cac1e4ebda0920d1f, State: Initialized, Role: FOLLOWER
I20260925 22:40:21.989045 28135 consensus_queue.cc:260] T 00000000000000000000000000000000 P 313b1c09ba64457cac1e4ebda0920d1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "313b1c09ba64457cac1e4ebda0920d1f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38861 } }
I20260925 22:40:21.989213 28135 raft_consensus.cc:388] T 00000000000000000000000000000000 P 313b1c09ba64457cac1e4ebda0920d1f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:40:21.989312 28135 raft_consensus.cc:482] T 00000000000000000000000000000000 P 313b1c09ba64457cac1e4ebda0920d1f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:40:21.989444 28135 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 313b1c09ba64457cac1e4ebda0920d1f [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:40:21.990545 28135 raft_consensus.cc:504] T 00000000000000000000000000000000 P 313b1c09ba64457cac1e4ebda0920d1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "313b1c09ba64457cac1e4ebda0920d1f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38861 } }
I20260925 22:40:21.990998 28135 leader_election.cc:302] T 00000000000000000000000000000000 P 313b1c09ba64457cac1e4ebda0920d1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 313b1c09ba64457cac1e4ebda0920d1f; no voters:
I20260925 22:40:21.991427 28135 leader_election.cc:288] T 00000000000000000000000000000000 P 313b1c09ba64457cac1e4ebda0920d1f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:40:21.991523 28140 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 313b1c09ba64457cac1e4ebda0920d1f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:40:21.991825 28140 raft_consensus.cc:686] T 00000000000000000000000000000000 P 313b1c09ba64457cac1e4ebda0920d1f [term 1 LEADER]: Becoming Leader. State: Replica: 313b1c09ba64457cac1e4ebda0920d1f, State: Running, Role: LEADER
I20260925 22:40:21.992244 28140 consensus_queue.cc:237] T 00000000000000000000000000000000 P 313b1c09ba64457cac1e4ebda0920d1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "313b1c09ba64457cac1e4ebda0920d1f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38861 } }
I20260925 22:40:21.993144 28135 sys_catalog.cc:564] T 00000000000000000000000000000000 P 313b1c09ba64457cac1e4ebda0920d1f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:40:21.994065 28138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:21.977985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.65:34347 (local address 127.0.0.1:38861)
0925 22:40:21.978591 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:21.978609 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:21.978651 (+ 42us) server_negotiation.cc:407] Connection header received
0925 22:40:21.979410 (+ 759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:21.979429 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:21.979842 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:21.980294 (+ 452us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:21.981265 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:21.982331 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:21.983341 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:21.983863 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:21.984059 (+ 196us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:21.987445 (+ 3386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:21.987453 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:21.987474 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:21.987512 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:21.989808 (+ 2296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:21.990390 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:21.990397 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:21.990404 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:21.990547 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:21.990908 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:21.990912 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:21.990914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:21.991479 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:21.991747 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:21.992627 (+ 880us) server_negotiation.cc:299] Negotiation successful
0925 22:40:21.992890 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":177,"threads_started":1}
I20260925 22:40:21.994354 28142 sys_catalog.cc:455] T 00000000000000000000000000000000 P 313b1c09ba64457cac1e4ebda0920d1f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 313b1c09ba64457cac1e4ebda0920d1f. Latest consensus state: current_term: 1 leader_uuid: "313b1c09ba64457cac1e4ebda0920d1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "313b1c09ba64457cac1e4ebda0920d1f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38861 } } }
I20260925 22:40:21.994478 28142 sys_catalog.cc:458] T 00000000000000000000000000000000 P 313b1c09ba64457cac1e4ebda0920d1f [sys.catalog]: This master's current role is: LEADER
I20260925 22:40:21.995479 28141 sys_catalog.cc:455] T 00000000000000000000000000000000 P 313b1c09ba64457cac1e4ebda0920d1f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "313b1c09ba64457cac1e4ebda0920d1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "313b1c09ba64457cac1e4ebda0920d1f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38861 } } }
I20260925 22:40:21.995582 28141 sys_catalog.cc:458] T 00000000000000000000000000000000 P 313b1c09ba64457cac1e4ebda0920d1f [sys.catalog]: This master's current role is: LEADER
I20260925 22:40:21.998096 7131 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260925 22:40:21.999207 28155 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 313b1c09ba64457cac1e4ebda0920d1f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:40:21.999338 28155 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:40:21.999419 28149 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:40:22.000391 28149 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:40:22.007916 28149 catalog_manager.cc:1347] Generated new cluster ID: 5c762332922440e786a46fe1ed38a5f0
I20260925 22:40:22.008006 28149 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:40:22.012537 28138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:22.002740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.65:45281 (local address 127.0.0.1:38861)
0925 22:40:22.002899 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:22.002903 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:22.002921 (+ 18us) server_negotiation.cc:407] Connection header received
0925 22:40:22.002969 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:22.002972 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:22.003061 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:22.003162 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:22.004191 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:22.005023 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:22.006107 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:22.006360 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:22.006477 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:22.008384 (+ 1907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:22.008388 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:22.008391 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:22.008418 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:22.010436 (+ 2018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:22.010997 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:22.011001 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:22.011003 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:22.011094 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:22.011447 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:22.011451 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:22.011452 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:22.011705 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:22.011859 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:22.012133 (+ 274us) server_negotiation.cc:299] Negotiation successful
0925 22:40:22.012243 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 22:40:22.023967 28149 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:40:22.025514 28149 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:40:22.039518 28149 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 313b1c09ba64457cac1e4ebda0920d1f: Generated new TSK 0
I20260925 22:40:22.040441 28149 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:40:22.052754 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 28067
W20260925 22:40:22.059810 28052 connection.cc:535] client connection to 127.0.0.1:38861 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38861): Connection reset by peer (error 104)
2026-09-25T22:40:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1497 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790375752797504-7131-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:40:22 dist-test-slave-wjdr krb5kdc[28166](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:40:22 dist-test-slave-wjdr krb5kdc[28166](info): set up 2 sockets
Sep 25 22:40:22 dist-test-slave-wjdr krb5kdc[28166](info): commencing operation
krb5kdc: starting...
W20260925 22:40:23.088945 7131 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Sep 25 22:40:23 dist-test-slave-wjdr krb5kdc[28166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376023, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:40:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:40:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790375752797504-7131-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790375752797504-7131-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 22:40:23.216284 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790375752797504-7131-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790375752797504-7131-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790375752797504-7131-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46623
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790375752797504-7131-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46623
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:40:23.333444 28179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:40:23.333724 28179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:40:23.333766 28179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:40:23.337884 28179 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:40:23.337941 28179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:40:23.337960 28179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:40:23.337996 28179 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:40:23.338027 28179 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:40:23.343382 28179 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790375752797504-7131-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790375752797504-7131-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46623
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790375752797504-7131-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28179
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790375752797504-7131-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:40:23.344491 28179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:40:23.345544 28179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:40:23.354182 28187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:23.354166 28185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:40:23.354492 28179 server_base.cc:1034] running on GCE node
W20260925 22:40:23.354259 28184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:40:23.355185 28179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:40:23.356344 28179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:40:23.357538 28179 hybrid_clock.cc:648] HybridClock initialized: now 1790376023357515 us; error 33 us; skew 500 ppm
Sep 25 22:40:23 dist-test-slave-wjdr krb5kdc[28166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376023, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:40:23.360312 28179 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 22:40:23.361285 28179 webserver.cc:460] Webserver started at http://127.0.0.1:33975/ using document root <none> and password file <none>
I20260925 22:40:23.361891 28179 fs_manager.cc:359] Metadata directory not provided
I20260925 22:40:23.361980 28179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:40:23.362228 28179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:40:23.364331 28179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790375752797504-7131-0/minicluster-data/master-0/data/instance:
uuid: "2cf16f261b204088a6e23599b90fcc4d"
format_stamp: "Formatted at 2026-09-25 22:40:23 on dist-test-slave-wjdr"
server_key: "d900485fe889a0ae27384dc98a7f2317"
server_key_iv: "10faca6660a143dc8591ad0862028fbc"
server_key_version: "encryptionkey@0"
I20260925 22:40:23.364972 28179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790375752797504-7131-0/minicluster-data/master-0/wal/instance:
uuid: "2cf16f261b204088a6e23599b90fcc4d"
format_stamp: "Formatted at 2026-09-25 22:40:23 on dist-test-slave-wjdr"
server_key: "d900485fe889a0ae27384dc98a7f2317"
server_key_iv: "10faca6660a143dc8591ad0862028fbc"
server_key_version: "encryptionkey@0"
I20260925 22:40:23.368708 28179 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:40:23.371285 28194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:23.372634 28179 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:40:23.372776 28179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790375752797504-7131-0/minicluster-data/master-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790375752797504-7131-0/minicluster-data/master-0/wal
uuid: "2cf16f261b204088a6e23599b90fcc4d"
format_stamp: "Formatted at 2026-09-25 22:40:23 on dist-test-slave-wjdr"
server_key: "d900485fe889a0ae27384dc98a7f2317"
server_key_iv: "10faca6660a143dc8591ad0862028fbc"
server_key_version: "encryptionkey@0"
I20260925 22:40:23.372895 28179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790375752797504-7131-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790375752797504-7131-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790375752797504-7131-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:40:23.387434 28179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:40:23.390410 28179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:40:23.390617 28179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:40:23.397488 28179 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46623
I20260925 22:40:23.397567 28246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46623 every 8 connection(s)
I20260925 22:40:23.398492 28179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
I20260925 22:40:23.401568 28247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:40:23.402621 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 28179
I20260925 22:40:23.402755 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790375752797504-7131-0/minicluster-data/master-0/wal/instance
I20260925 22:40:23.403101 7131 external_mini_cluster.cc:1428] Setting key f32a6275c2a38a840d1267e3a055093d
I20260925 22:40:23.407977 28247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cf16f261b204088a6e23599b90fcc4d: Bootstrap starting.
I20260925 22:40:23.410681 28247 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2cf16f261b204088a6e23599b90fcc4d: Neither blocks nor log segments found. Creating new log.
I20260925 22:40:23.411566 28247 log.cc:824] T 00000000000000000000000000000000 P 2cf16f261b204088a6e23599b90fcc4d: Log is configured to *not* fsync() on all Append() calls
I20260925 22:40:23.412209 28250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:23.404558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.65:45365 (local address 127.0.0.1:46623)
0925 22:40:23.404999 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:23.405014 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:23.405049 (+ 35us) server_negotiation.cc:407] Connection header received
0925 22:40:23.405670 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:23.405686 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:23.406010 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:23.406275 (+ 265us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:23.406639 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:23.407606 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:23.408828 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:23.409285 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:23.409420 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:23.410280 (+ 860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:23.410285 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:23.410298 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:40:23.410706 (+ 408us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:23.410758 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:40:23.410808 (+ 50us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:23.410941 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:23.411140 (+ 199us) server_negotiation.cc:299] Negotiation successful
0925 22:40:23.411302 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":125,"threads_started":1}
I20260925 22:40:23.413616 28247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cf16f261b204088a6e23599b90fcc4d: No bootstrap required, opened a new log
I20260925 22:40:23.415115 7131 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:40:23.418123 28247 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2cf16f261b204088a6e23599b90fcc4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cf16f261b204088a6e23599b90fcc4d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46623 } }
I20260925 22:40:23.418422 28247 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2cf16f261b204088a6e23599b90fcc4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:40:23.418511 28247 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2cf16f261b204088a6e23599b90fcc4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cf16f261b204088a6e23599b90fcc4d, State: Initialized, Role: FOLLOWER
I20260925 22:40:23.418951 28247 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2cf16f261b204088a6e23599b90fcc4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cf16f261b204088a6e23599b90fcc4d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46623 } }
I20260925 22:40:23.419101 28247 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2cf16f261b204088a6e23599b90fcc4d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:40:23.419170 28247 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2cf16f261b204088a6e23599b90fcc4d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:40:23.419269 28247 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2cf16f261b204088a6e23599b90fcc4d [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:40:23.420202 28247 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2cf16f261b204088a6e23599b90fcc4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cf16f261b204088a6e23599b90fcc4d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46623 } }
I20260925 22:40:23.420626 28247 leader_election.cc:302] T 00000000000000000000000000000000 P 2cf16f261b204088a6e23599b90fcc4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2cf16f261b204088a6e23599b90fcc4d; no voters:
I20260925 22:40:23.420989 28247 leader_election.cc:288] T 00000000000000000000000000000000 P 2cf16f261b204088a6e23599b90fcc4d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:40:23.421061 28257 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2cf16f261b204088a6e23599b90fcc4d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:40:23.421263 28257 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2cf16f261b204088a6e23599b90fcc4d [term 1 LEADER]: Becoming Leader. State: Replica: 2cf16f261b204088a6e23599b90fcc4d, State: Running, Role: LEADER
I20260925 22:40:23.421566 28257 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2cf16f261b204088a6e23599b90fcc4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cf16f261b204088a6e23599b90fcc4d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46623 } }
I20260925 22:40:23.422194 28250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:23.417657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.65:43075 (local address 127.0.0.1:46623)
0925 22:40:23.417795 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:23.417800 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:23.418007 (+ 207us) server_negotiation.cc:407] Connection header received
0925 22:40:23.418106 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:23.418109 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:23.418162 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:23.418245 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:23.418657 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:23.419337 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:23.420163 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:23.420604 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:23.420716 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:23.421662 (+ 946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:23.421665 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:23.421668 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:40:23.421681 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:23.421709 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:40:23.421721 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:23.421787 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:23.422002 (+ 215us) server_negotiation.cc:299] Negotiation successful
0925 22:40:23.422039 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 22:40:23.422335 28247 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2cf16f261b204088a6e23599b90fcc4d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:40:23.423358 28259 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cf16f261b204088a6e23599b90fcc4d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2cf16f261b204088a6e23599b90fcc4d. Latest consensus state: current_term: 1 leader_uuid: "2cf16f261b204088a6e23599b90fcc4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cf16f261b204088a6e23599b90fcc4d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46623 } } }
I20260925 22:40:23.423487 28259 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cf16f261b204088a6e23599b90fcc4d [sys.catalog]: This master's current role is: LEADER
I20260925 22:40:23.423878 28266 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:40:23.423831 28258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cf16f261b204088a6e23599b90fcc4d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2cf16f261b204088a6e23599b90fcc4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cf16f261b204088a6e23599b90fcc4d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46623 } } }
I20260925 22:40:23.424005 28258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cf16f261b204088a6e23599b90fcc4d [sys.catalog]: This master's current role is: LEADER
I20260925 22:40:23.427101 28266 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:40:23.432420 28266 catalog_manager.cc:1347] Generated new cluster ID: 269bf0b532474efcafefccc952d13c58
I20260925 22:40:23.432518 28266 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:40:23.445467 28266 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:40:23.446750 28266 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:40:23.452888 28266 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2cf16f261b204088a6e23599b90fcc4d: Generated new TSK 0
I20260925 22:40:23.453518 28266 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:40:23.458963 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 28179
W20260925 22:40:23.464325 28164 connection.cc:535] client connection to 127.0.0.1:46623 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46623): Connection reset by peer (error 104)
2026-09-25T22:40:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1403 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790375752797504-7131-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:40:23 dist-test-slave-wjdr krb5kdc[28278](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:40:23 dist-test-slave-wjdr krb5kdc[28278](info): set up 2 sockets
Sep 25 22:40:23 dist-test-slave-wjdr krb5kdc[28278](info): commencing operation
krb5kdc: starting...
W20260925 22:40:24.492722 7131 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Sep 25 22:40:24 dist-test-slave-wjdr krb5kdc[28278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:40:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:40:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790375752797504-7131-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790375752797504-7131-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 22:40:24.611984 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790375752797504-7131-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790375752797504-7131-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790375752797504-7131-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38305
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790375752797504-7131-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38305
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:40:24.720443 28291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:40:24.720713 28291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:40:24.720754 28291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:40:24.724757 28291 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:40:24.724835 28291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:40:24.724854 28291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:40:24.724870 28291 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:40:24.724943 28291 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:40:24.730055 28291 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790375752797504-7131-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790375752797504-7131-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38305
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790375752797504-7131-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28291
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790375752797504-7131-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:40:24.731097 28291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:40:24.732095 28291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:40:24.740824 28299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:24.740846 28296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:40:24.740906 28291 server_base.cc:1034] running on GCE node
W20260925 22:40:24.740832 28297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:40:24.741585 28291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:40:24.742657 28291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:40:24.743822 28291 hybrid_clock.cc:648] HybridClock initialized: now 1790376024743804 us; error 33 us; skew 500 ppm
Sep 25 22:40:24 dist-test-slave-wjdr krb5kdc[28278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376024, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:40:24.746567 28291 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 22:40:24.747445 28291 webserver.cc:460] Webserver started at http://127.0.0.1:33055/ using document root <none> and password file <none>
I20260925 22:40:24.747949 28291 fs_manager.cc:359] Metadata directory not provided
I20260925 22:40:24.748005 28291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:40:24.748160 28291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:40:24.750142 28291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790375752797504-7131-0/minicluster-data/master-0/data/instance:
uuid: "a83cab403c2f4751b63482b1ebdaed94"
format_stamp: "Formatted at 2026-09-25 22:40:24 on dist-test-slave-wjdr"
server_key: "0eb47f5613d36abdb84490f5d705d98b"
server_key_iv: "bc335358aabbbfb84d0268935ec0fbb9"
server_key_version: "encryptionkey@0"
I20260925 22:40:24.750644 28291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790375752797504-7131-0/minicluster-data/master-0/wal/instance:
uuid: "a83cab403c2f4751b63482b1ebdaed94"
format_stamp: "Formatted at 2026-09-25 22:40:24 on dist-test-slave-wjdr"
server_key: "0eb47f5613d36abdb84490f5d705d98b"
server_key_iv: "bc335358aabbbfb84d0268935ec0fbb9"
server_key_version: "encryptionkey@0"
I20260925 22:40:24.754338 28291 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260925 22:40:24.756703 28306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:24.757949 28291 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 22:40:24.758074 28291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790375752797504-7131-0/minicluster-data/master-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790375752797504-7131-0/minicluster-data/master-0/wal
uuid: "a83cab403c2f4751b63482b1ebdaed94"
format_stamp: "Formatted at 2026-09-25 22:40:24 on dist-test-slave-wjdr"
server_key: "0eb47f5613d36abdb84490f5d705d98b"
server_key_iv: "bc335358aabbbfb84d0268935ec0fbb9"
server_key_version: "encryptionkey@0"
I20260925 22:40:24.758183 28291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790375752797504-7131-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790375752797504-7131-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790375752797504-7131-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:40:24.764096 28291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:40:24.767434 28291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:40:24.767630 28291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:40:24.774638 28291 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38305
I20260925 22:40:24.774662 28358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38305 every 8 connection(s)
I20260925 22:40:24.775539 28291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
I20260925 22:40:24.777704 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 28291
I20260925 22:40:24.777843 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790375752797504-7131-0/minicluster-data/master-0/wal/instance
I20260925 22:40:24.778133 7131 external_mini_cluster.cc:1428] Setting key 249e557c39f94097926ebadffd2ff3a1
I20260925 22:40:24.778281 28359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:40:24.783836 28362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:24.779639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.65:54875 (local address 127.0.0.1:38305)
0925 22:40:24.780073 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:24.780102 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:24.780143 (+ 41us) server_negotiation.cc:407] Connection header received
0925 22:40:24.781057 (+ 914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:24.781080 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:24.781443 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:24.781812 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:24.782483 (+ 671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:24.782489 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:24.782503 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:40:24.782513 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:24.782571 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:40:24.782596 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:24.782731 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:24.782806 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 22:40:24.782955 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":120,"threads_started":1}
I20260925 22:40:24.785635 28359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a83cab403c2f4751b63482b1ebdaed94: Bootstrap starting.
I20260925 22:40:24.786562 7131 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:40:24.787943 28359 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a83cab403c2f4751b63482b1ebdaed94: Neither blocks nor log segments found. Creating new log.
I20260925 22:40:24.788807 28359 log.cc:824] T 00000000000000000000000000000000 P a83cab403c2f4751b63482b1ebdaed94: Log is configured to *not* fsync() on all Append() calls
I20260925 22:40:24.790325 28362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:24.788535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.65:52327 (local address 127.0.0.1:38305)
0925 22:40:24.788665 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:24.788670 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:24.788886 (+ 216us) server_negotiation.cc:407] Connection header received
0925 22:40:24.789013 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:24.789015 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:24.789043 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:24.789106 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:24.789907 (+ 801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:24.789910 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:24.789912 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:40:24.789921 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:24.789944 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:40:24.789954 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:24.790019 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:24.790126 (+ 107us) server_negotiation.cc:299] Negotiation successful
0925 22:40:24.790157 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 22:40:24.791002 28359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a83cab403c2f4751b63482b1ebdaed94: No bootstrap required, opened a new log
I20260925 22:40:24.794448 28359 raft_consensus.cc:348] T 00000000000000000000000000000000 P a83cab403c2f4751b63482b1ebdaed94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a83cab403c2f4751b63482b1ebdaed94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38305 } }
I20260925 22:40:24.794682 28359 raft_consensus.cc:374] T 00000000000000000000000000000000 P a83cab403c2f4751b63482b1ebdaed94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:40:24.794744 28359 raft_consensus.cc:729] T 00000000000000000000000000000000 P a83cab403c2f4751b63482b1ebdaed94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a83cab403c2f4751b63482b1ebdaed94, State: Initialized, Role: FOLLOWER
I20260925 22:40:24.795176 28359 consensus_queue.cc:260] T 00000000000000000000000000000000 P a83cab403c2f4751b63482b1ebdaed94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a83cab403c2f4751b63482b1ebdaed94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38305 } }
I20260925 22:40:24.795328 28359 raft_consensus.cc:388] T 00000000000000000000000000000000 P a83cab403c2f4751b63482b1ebdaed94 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:40:24.795395 28359 raft_consensus.cc:482] T 00000000000000000000000000000000 P a83cab403c2f4751b63482b1ebdaed94 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:40:24.795487 28359 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a83cab403c2f4751b63482b1ebdaed94 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:40:24.796341 28359 raft_consensus.cc:504] T 00000000000000000000000000000000 P a83cab403c2f4751b63482b1ebdaed94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a83cab403c2f4751b63482b1ebdaed94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38305 } }
I20260925 22:40:24.796689 28359 leader_election.cc:302] T 00000000000000000000000000000000 P a83cab403c2f4751b63482b1ebdaed94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a83cab403c2f4751b63482b1ebdaed94; no voters:
I20260925 22:40:24.796979 28359 leader_election.cc:288] T 00000000000000000000000000000000 P a83cab403c2f4751b63482b1ebdaed94 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:40:24.797142 28369 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a83cab403c2f4751b63482b1ebdaed94 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:40:24.797369 28369 raft_consensus.cc:686] T 00000000000000000000000000000000 P a83cab403c2f4751b63482b1ebdaed94 [term 1 LEADER]: Becoming Leader. State: Replica: a83cab403c2f4751b63482b1ebdaed94, State: Running, Role: LEADER
I20260925 22:40:24.797690 28369 consensus_queue.cc:237] T 00000000000000000000000000000000 P a83cab403c2f4751b63482b1ebdaed94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a83cab403c2f4751b63482b1ebdaed94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38305 } }
I20260925 22:40:24.798168 28359 sys_catalog.cc:564] T 00000000000000000000000000000000 P a83cab403c2f4751b63482b1ebdaed94 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:40:24.799173 28370 sys_catalog.cc:455] T 00000000000000000000000000000000 P a83cab403c2f4751b63482b1ebdaed94 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a83cab403c2f4751b63482b1ebdaed94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a83cab403c2f4751b63482b1ebdaed94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38305 } } }
I20260925 22:40:24.799299 28370 sys_catalog.cc:458] T 00000000000000000000000000000000 P a83cab403c2f4751b63482b1ebdaed94 [sys.catalog]: This master's current role is: LEADER
I20260925 22:40:24.799665 28378 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:40:24.799949 28371 sys_catalog.cc:455] T 00000000000000000000000000000000 P a83cab403c2f4751b63482b1ebdaed94 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a83cab403c2f4751b63482b1ebdaed94. Latest consensus state: current_term: 1 leader_uuid: "a83cab403c2f4751b63482b1ebdaed94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a83cab403c2f4751b63482b1ebdaed94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38305 } } }
I20260925 22:40:24.800076 28371 sys_catalog.cc:458] T 00000000000000000000000000000000 P a83cab403c2f4751b63482b1ebdaed94 [sys.catalog]: This master's current role is: LEADER
I20260925 22:40:24.802584 28378 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:40:24.807904 28378 catalog_manager.cc:1347] Generated new cluster ID: 99cad0fe662d47bfa8e4d153e3eebc13
I20260925 22:40:24.807973 28378 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:40:24.825646 28378 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:40:24.826030 28378 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:40:24.834889 28378 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a83cab403c2f4751b63482b1ebdaed94: Generated new TSK 0
I20260925 22:40:24.835579 28378 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:40:24.843891 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 28291
2026-09-25T22:40:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1384 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790375752797504-7131-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:40:24 dist-test-slave-wjdr krb5kdc[28390](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:40:24 dist-test-slave-wjdr krb5kdc[28390](info): set up 2 sockets
Sep 25 22:40:24 dist-test-slave-wjdr krb5kdc[28390](info): commencing operation
krb5kdc: starting...
W20260925 22:40:25.889333 7131 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Sep 25 22:40:25 dist-test-slave-wjdr krb5kdc[28390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376025, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:40:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:40:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790375752797504-7131-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790375752797504-7131-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 22:40:26.023382 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790375752797504-7131-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790375752797504-7131-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790375752797504-7131-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32779
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790375752797504-7131-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32779
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:40:26.157732 28403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:40:26.158118 28403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:40:26.158200 28403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:40:26.162163 28403 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:40:26.162254 28403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:40:26.162292 28403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:40:26.162350 28403 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:40:26.162390 28403 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:40:26.167735 28403 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790375752797504-7131-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790375752797504-7131-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32779
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790375752797504-7131-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28403
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790375752797504-7131-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:40:26.169138 28403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:40:26.170352 28403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:40:26.179939 28408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:26.179934 28411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:26.179986 28409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:40:26.180181 28403 server_base.cc:1034] running on GCE node
I20260925 22:40:26.181037 28403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:40:26.182318 28403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:40:26.183483 28403 hybrid_clock.cc:648] HybridClock initialized: now 1790376026183456 us; error 45 us; skew 500 ppm
Sep 25 22:40:26 dist-test-slave-wjdr krb5kdc[28390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376026, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:40:26.190588 28403 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 22:40:26.191838 28403 webserver.cc:460] Webserver started at http://127.0.0.1:37667/ using document root <none> and password file <none>
I20260925 22:40:26.192606 28403 fs_manager.cc:359] Metadata directory not provided
I20260925 22:40:26.192687 28403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:40:26.192911 28403 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:40:26.195494 28403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790375752797504-7131-0/minicluster-data/master-0/data/instance:
uuid: "17754678527a445c9f81913e016357f4"
format_stamp: "Formatted at 2026-09-25 22:40:26 on dist-test-slave-wjdr"
server_key: "191929cceb76fcd86009b53d6482fdda"
server_key_iv: "a73ad91dd61c181a4fb6067537dd45ae"
server_key_version: "encryptionkey@0"
I20260925 22:40:26.196103 28403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790375752797504-7131-0/minicluster-data/master-0/wal/instance:
uuid: "17754678527a445c9f81913e016357f4"
format_stamp: "Formatted at 2026-09-25 22:40:26 on dist-test-slave-wjdr"
server_key: "191929cceb76fcd86009b53d6482fdda"
server_key_iv: "a73ad91dd61c181a4fb6067537dd45ae"
server_key_version: "encryptionkey@0"
I20260925 22:40:26.200700 28403 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260925 22:40:26.203625 28418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:26.205155 28403 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:40:26.205308 28403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790375752797504-7131-0/minicluster-data/master-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790375752797504-7131-0/minicluster-data/master-0/wal
uuid: "17754678527a445c9f81913e016357f4"
format_stamp: "Formatted at 2026-09-25 22:40:26 on dist-test-slave-wjdr"
server_key: "191929cceb76fcd86009b53d6482fdda"
server_key_iv: "a73ad91dd61c181a4fb6067537dd45ae"
server_key_version: "encryptionkey@0"
I20260925 22:40:26.205427 28403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790375752797504-7131-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790375752797504-7131-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790375752797504-7131-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:40:26.221946 28403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:40:26.225721 28403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:40:26.225961 28403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:40:26.234210 28403 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32779
I20260925 22:40:26.234236 28470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32779 every 8 connection(s)
I20260925 22:40:26.235337 28403 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
I20260925 22:40:26.238773 28471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:40:26.241889 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 28403
I20260925 22:40:26.242093 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790375752797504-7131-0/minicluster-data/master-0/wal/instance
I20260925 22:40:26.242458 7131 external_mini_cluster.cc:1428] Setting key 333303e6c15cd6f24a239f174ea8d7f0
I20260925 22:40:26.246249 28471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17754678527a445c9f81913e016357f4: Bootstrap starting.
I20260925 22:40:26.249460 28471 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17754678527a445c9f81913e016357f4: Neither blocks nor log segments found. Creating new log.
I20260925 22:40:26.250473 28471 log.cc:824] T 00000000000000000000000000000000 P 17754678527a445c9f81913e016357f4: Log is configured to *not* fsync() on all Append() calls
I20260925 22:40:26.252856 28471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17754678527a445c9f81913e016357f4: No bootstrap required, opened a new log
Sep 25 22:40:26 dist-test-slave-wjdr krb5kdc[28390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376025, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 22:40:26.257236 28471 raft_consensus.cc:348] T 00000000000000000000000000000000 P 17754678527a445c9f81913e016357f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17754678527a445c9f81913e016357f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32779 } }
I20260925 22:40:26.257513 28471 raft_consensus.cc:374] T 00000000000000000000000000000000 P 17754678527a445c9f81913e016357f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:40:26.257602 28471 raft_consensus.cc:729] T 00000000000000000000000000000000 P 17754678527a445c9f81913e016357f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17754678527a445c9f81913e016357f4, State: Initialized, Role: FOLLOWER
I20260925 22:40:26.258216 28471 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17754678527a445c9f81913e016357f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17754678527a445c9f81913e016357f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32779 } }
I20260925 22:40:26.258364 28471 raft_consensus.cc:388] T 00000000000000000000000000000000 P 17754678527a445c9f81913e016357f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:40:26.258455 28471 raft_consensus.cc:482] T 00000000000000000000000000000000 P 17754678527a445c9f81913e016357f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:40:26.258571 28471 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 17754678527a445c9f81913e016357f4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:40:26.259259 28474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:26.244202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.65:51487 (local address 127.0.0.1:32779)
0925 22:40:26.244746 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:26.244770 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:26.244820 (+ 50us) server_negotiation.cc:407] Connection header received
0925 22:40:26.245710 (+ 890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:26.245734 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:26.246206 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:26.246610 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:26.247345 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:26.248936 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:26.249975 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:26.250707 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:26.250883 (+ 176us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:26.254034 (+ 3151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:26.254040 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:26.254055 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:26.254081 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:26.255878 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:26.256502 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:26.256515 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:26.256520 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:26.256631 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:26.256968 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:26.256971 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:26.256973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:26.257374 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:26.257610 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:26.258001 (+ 391us) server_negotiation.cc:299] Negotiation successful
0925 22:40:26.258273 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":182,"threads_started":1}
I20260925 22:40:26.259675 28471 raft_consensus.cc:504] T 00000000000000000000000000000000 P 17754678527a445c9f81913e016357f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17754678527a445c9f81913e016357f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32779 } }
I20260925 22:40:26.260047 28471 leader_election.cc:302] T 00000000000000000000000000000000 P 17754678527a445c9f81913e016357f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 17754678527a445c9f81913e016357f4; no voters:
I20260925 22:40:26.260428 28471 leader_election.cc:288] T 00000000000000000000000000000000 P 17754678527a445c9f81913e016357f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:40:26.260608 28476 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 17754678527a445c9f81913e016357f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:40:26.260816 28476 raft_consensus.cc:686] T 00000000000000000000000000000000 P 17754678527a445c9f81913e016357f4 [term 1 LEADER]: Becoming Leader. State: Replica: 17754678527a445c9f81913e016357f4, State: Running, Role: LEADER
I20260925 22:40:26.261196 28476 consensus_queue.cc:237] T 00000000000000000000000000000000 P 17754678527a445c9f81913e016357f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17754678527a445c9f81913e016357f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32779 } }
I20260925 22:40:26.261919 28471 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17754678527a445c9f81913e016357f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:40:26.262908 7131 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:40:26.263125 28478 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17754678527a445c9f81913e016357f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17754678527a445c9f81913e016357f4. Latest consensus state: current_term: 1 leader_uuid: "17754678527a445c9f81913e016357f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17754678527a445c9f81913e016357f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32779 } } }
I20260925 22:40:26.263276 28478 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17754678527a445c9f81913e016357f4 [sys.catalog]: This master's current role is: LEADER
I20260925 22:40:26.263679 28485 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:40:26.264446 28477 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17754678527a445c9f81913e016357f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "17754678527a445c9f81913e016357f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17754678527a445c9f81913e016357f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32779 } } }
I20260925 22:40:26.264734 28477 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17754678527a445c9f81913e016357f4 [sys.catalog]: This master's current role is: LEADER
I20260925 22:40:26.267330 28485 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:40:26.274493 28485 catalog_manager.cc:1347] Generated new cluster ID: d8b9aa989fc348ec8aac1142edf49802
I20260925 22:40:26.274554 28485 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:40:26.275417 28474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:26.265934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.65:39057 (local address 127.0.0.1:32779)
0925 22:40:26.266042 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:26.266047 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:26.266606 (+ 559us) server_negotiation.cc:407] Connection header received
0925 22:40:26.266733 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:26.266737 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:26.266792 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:26.266885 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:26.267843 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:26.268635 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:26.269464 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:26.269729 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:26.269841 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:26.271376 (+ 1535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:26.271380 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:26.271384 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:26.271409 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:26.273429 (+ 2020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:26.273935 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:26.273938 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:26.273940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:26.274021 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:26.274357 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:26.274360 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:26.274362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:26.274591 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:26.274749 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:26.275154 (+ 405us) server_negotiation.cc:299] Negotiation successful
0925 22:40:26.275256 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 22:40:26.297392 28485 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:40:26.298592 28485 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:40:26.303490 28485 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 17754678527a445c9f81913e016357f4: Generated new TSK 0
I20260925 22:40:26.304270 28485 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:40:26.312388 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 28403
W20260925 22:40:26.320797 28388 connection.cc:535] client connection to 127.0.0.1:32779 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32779): Connection reset by peer (error 104)
2026-09-25T22:40:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1472 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790375752797504-7131-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:40:26 dist-test-slave-wjdr krb5kdc[28502](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:40:26 dist-test-slave-wjdr krb5kdc[28502](info): set up 2 sockets
Sep 25 22:40:26 dist-test-slave-wjdr krb5kdc[28502](info): commencing operation
krb5kdc: starting...
W20260925 22:40:27.356989 7131 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Sep 25 22:40:27 dist-test-slave-wjdr krb5kdc[28502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:40:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:40:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790375752797504-7131-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790375752797504-7131-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 22:40:27.479013 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790375752797504-7131-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790375752797504-7131-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790375752797504-7131-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40299
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790375752797504-7131-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40299
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:40:27.596306 28515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:40:27.596630 28515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:40:27.596714 28515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:40:27.600749 28515 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:40:27.600857 28515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:40:27.600894 28515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:40:27.600953 28515 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:40:27.601006 28515 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:40:27.606488 28515 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790375752797504-7131-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790375752797504-7131-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40299
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790375752797504-7131-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28515
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790375752797504-7131-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:40:27.607650 28515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:40:27.608732 28515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:40:27.617455 28521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:27.617623 28520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:40:27.617765 28515 server_base.cc:1034] running on GCE node
W20260925 22:40:27.617456 28523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:40:27.618460 28515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:40:27.619616 28515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:40:27.620801 28515 hybrid_clock.cc:648] HybridClock initialized: now 1790376027620778 us; error 46 us; skew 500 ppm
Sep 25 22:40:27 dist-test-slave-wjdr krb5kdc[28502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376027, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:40:27.623466 28515 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 22:40:27.624317 28515 webserver.cc:460] Webserver started at http://127.0.0.1:38047/ using document root <none> and password file <none>
I20260925 22:40:27.624941 28515 fs_manager.cc:359] Metadata directory not provided
I20260925 22:40:27.625003 28515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:40:27.625173 28515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:40:27.627168 28515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790375752797504-7131-0/minicluster-data/master-0/data/instance:
uuid: "4e52e4f046c94fe3b2b93208a91d1c7a"
format_stamp: "Formatted at 2026-09-25 22:40:27 on dist-test-slave-wjdr"
server_key: "5e63b55b07535356d074db9b9b5f30a4"
server_key_iv: "391230da15d7f45e0902f25dfbc4e79f"
server_key_version: "encryptionkey@0"
I20260925 22:40:27.627671 28515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790375752797504-7131-0/minicluster-data/master-0/wal/instance:
uuid: "4e52e4f046c94fe3b2b93208a91d1c7a"
format_stamp: "Formatted at 2026-09-25 22:40:27 on dist-test-slave-wjdr"
server_key: "5e63b55b07535356d074db9b9b5f30a4"
server_key_iv: "391230da15d7f45e0902f25dfbc4e79f"
server_key_version: "encryptionkey@0"
I20260925 22:40:27.631510 28515 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260925 22:40:27.634052 28530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:27.635362 28515 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260925 22:40:27.635495 28515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790375752797504-7131-0/minicluster-data/master-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790375752797504-7131-0/minicluster-data/master-0/wal
uuid: "4e52e4f046c94fe3b2b93208a91d1c7a"
format_stamp: "Formatted at 2026-09-25 22:40:27 on dist-test-slave-wjdr"
server_key: "5e63b55b07535356d074db9b9b5f30a4"
server_key_iv: "391230da15d7f45e0902f25dfbc4e79f"
server_key_version: "encryptionkey@0"
I20260925 22:40:27.635607 28515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790375752797504-7131-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790375752797504-7131-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790375752797504-7131-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:40:27.647477 28515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:40:27.650723 28515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:40:27.650964 28515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:40:27.658223 28515 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40299
I20260925 22:40:27.658281 28582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40299 every 8 connection(s)
I20260925 22:40:27.659230 28515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
I20260925 22:40:27.662197 28583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:40:27.665409 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 28515
I20260925 22:40:27.665560 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790375752797504-7131-0/minicluster-data/master-0/wal/instance
I20260925 22:40:27.665853 7131 external_mini_cluster.cc:1428] Setting key 74499f712d79797cfa5ef1b1b1751a8e
I20260925 22:40:27.668660 28583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e52e4f046c94fe3b2b93208a91d1c7a: Bootstrap starting.
I20260925 22:40:27.671495 28583 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e52e4f046c94fe3b2b93208a91d1c7a: Neither blocks nor log segments found. Creating new log.
I20260925 22:40:27.672605 28583 log.cc:824] T 00000000000000000000000000000000 P 4e52e4f046c94fe3b2b93208a91d1c7a: Log is configured to *not* fsync() on all Append() calls
I20260925 22:40:27.674502 28583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e52e4f046c94fe3b2b93208a91d1c7a: No bootstrap required, opened a new log
I20260925 22:40:27.675863 28586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:27.667349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.65:37167 (local address 127.0.0.1:40299)
0925 22:40:27.667780 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:27.667800 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:27.667843 (+ 43us) server_negotiation.cc:407] Connection header received
0925 22:40:27.668671 (+ 828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:27.668694 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:27.669114 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:27.669477 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:27.669898 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:27.671420 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:27.672339 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:27.673067 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:27.673244 (+ 177us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:27.674166 (+ 922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:27.674172 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:27.674195 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:40:27.674589 (+ 394us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:27.674682 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:40:27.674718 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:27.674858 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:27.675010 (+ 152us) server_negotiation.cc:299] Negotiation successful
0925 22:40:27.675157 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":111,"threads_started":1}
I20260925 22:40:27.678303 28583 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e52e4f046c94fe3b2b93208a91d1c7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e52e4f046c94fe3b2b93208a91d1c7a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40299 } }
I20260925 22:40:27.678542 28583 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e52e4f046c94fe3b2b93208a91d1c7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:40:27.678630 28583 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e52e4f046c94fe3b2b93208a91d1c7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e52e4f046c94fe3b2b93208a91d1c7a, State: Initialized, Role: FOLLOWER
I20260925 22:40:27.678975 7131 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:40:27.679116 28583 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e52e4f046c94fe3b2b93208a91d1c7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e52e4f046c94fe3b2b93208a91d1c7a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40299 } }
I20260925 22:40:27.679258 28583 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e52e4f046c94fe3b2b93208a91d1c7a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:40:27.679315 28583 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e52e4f046c94fe3b2b93208a91d1c7a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:40:27.679404 28583 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e52e4f046c94fe3b2b93208a91d1c7a [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:40:27.680564 28583 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e52e4f046c94fe3b2b93208a91d1c7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e52e4f046c94fe3b2b93208a91d1c7a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40299 } }
I20260925 22:40:27.680939 28583 leader_election.cc:302] T 00000000000000000000000000000000 P 4e52e4f046c94fe3b2b93208a91d1c7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e52e4f046c94fe3b2b93208a91d1c7a; no voters:
I20260925 22:40:27.681330 28583 leader_election.cc:288] T 00000000000000000000000000000000 P 4e52e4f046c94fe3b2b93208a91d1c7a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:40:27.681412 28592 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e52e4f046c94fe3b2b93208a91d1c7a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:40:27.681646 28592 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e52e4f046c94fe3b2b93208a91d1c7a [term 1 LEADER]: Becoming Leader. State: Replica: 4e52e4f046c94fe3b2b93208a91d1c7a, State: Running, Role: LEADER
I20260925 22:40:27.682081 28592 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e52e4f046c94fe3b2b93208a91d1c7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e52e4f046c94fe3b2b93208a91d1c7a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40299 } }
I20260925 22:40:27.682695 28583 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e52e4f046c94fe3b2b93208a91d1c7a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:40:27.683261 28592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e52e4f046c94fe3b2b93208a91d1c7a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e52e4f046c94fe3b2b93208a91d1c7a. Latest consensus state: current_term: 1 leader_uuid: "4e52e4f046c94fe3b2b93208a91d1c7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e52e4f046c94fe3b2b93208a91d1c7a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40299 } } }
I20260925 22:40:27.683986 28592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e52e4f046c94fe3b2b93208a91d1c7a [sys.catalog]: This master's current role is: LEADER
I20260925 22:40:27.683874 28594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e52e4f046c94fe3b2b93208a91d1c7a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4e52e4f046c94fe3b2b93208a91d1c7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e52e4f046c94fe3b2b93208a91d1c7a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40299 } } }
I20260925 22:40:27.684315 28594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e52e4f046c94fe3b2b93208a91d1c7a [sys.catalog]: This master's current role is: LEADER
I20260925 22:40:27.684649 28603 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:40:27.686506 28586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:27.681369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.65:58003 (local address 127.0.0.1:40299)
0925 22:40:27.681485 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:27.681489 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:27.681742 (+ 253us) server_negotiation.cc:407] Connection header received
0925 22:40:27.681936 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:27.681940 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:27.681988 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:27.682073 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:27.682423 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:27.683142 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:27.684420 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:27.684803 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:27.684902 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:27.686018 (+ 1116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:27.686022 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:27.686025 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:40:27.686037 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:27.686071 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:40:27.686082 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:27.686179 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:27.686316 (+ 137us) server_negotiation.cc:299] Negotiation successful
0925 22:40:27.686352 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 22:40:27.688000 28603 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:40:27.693632 28603 catalog_manager.cc:1347] Generated new cluster ID: 386f5d9f4bf74eccb182002cf3451549
I20260925 22:40:27.693696 28603 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:40:27.703734 28603 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:40:27.704816 28603 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:40:27.711838 28603 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e52e4f046c94fe3b2b93208a91d1c7a: Generated new TSK 0
I20260925 22:40:27.712769 28603 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:40:27.722911 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 28515
W20260925 22:40:27.728230 28500 connection.cc:535] client connection to 127.0.0.1:40299 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40299): Connection reset by peer (error 104)
2026-09-25T22:40:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1406 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790375752797504-7131-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:40:27 dist-test-slave-wjdr krb5kdc[28614](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:40:27 dist-test-slave-wjdr krb5kdc[28614](info): set up 2 sockets
Sep 25 22:40:27 dist-test-slave-wjdr krb5kdc[28614](info): commencing operation
krb5kdc: starting...
W20260925 22:40:28.765070 7131 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Sep 25 22:40:28 dist-test-slave-wjdr krb5kdc[28614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:40:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:40:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790375752797504-7131-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790375752797504-7131-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 22:40:28.919602 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790375752797504-7131-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790375752797504-7131-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790375752797504-7131-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38539
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790375752797504-7131-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38539
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:40:29.032790 28627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:40:29.033119 28627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:40:29.033197 28627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:40:29.037266 28627 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:40:29.037356 28627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:40:29.037392 28627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:40:29.037449 28627 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:40:29.037485 28627 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:40:29.042743 28627 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790375752797504-7131-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790375752797504-7131-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38539
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790375752797504-7131-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790375752797504-7131-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:40:29.043867 28627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:40:29.044905 28627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:40:29.053478 28635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:29.053452 28632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:29.053494 28633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:40:29.053694 28627 server_base.cc:1034] running on GCE node
I20260925 22:40:29.054394 28627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:40:29.055521 28627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:40:29.056710 28627 hybrid_clock.cc:648] HybridClock initialized: now 1790376029056687 us; error 51 us; skew 500 ppm
Sep 25 22:40:29 dist-test-slave-wjdr krb5kdc[28614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376029, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:40:29.059510 28627 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 22:40:29.060420 28627 webserver.cc:460] Webserver started at http://127.0.0.1:46199/ using document root <none> and password file <none>
I20260925 22:40:29.061003 28627 fs_manager.cc:359] Metadata directory not provided
I20260925 22:40:29.061098 28627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:40:29.061336 28627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:40:29.063366 28627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790375752797504-7131-0/minicluster-data/master-0/data/instance:
uuid: "0dbc15db9893492a8759b2a8e30ce938"
format_stamp: "Formatted at 2026-09-25 22:40:29 on dist-test-slave-wjdr"
server_key: "3082f99b89eca289f373707e8e922511"
server_key_iv: "5aaf227008438ad43d7a3320e20868f3"
server_key_version: "encryptionkey@0"
I20260925 22:40:29.063925 28627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790375752797504-7131-0/minicluster-data/master-0/wal/instance:
uuid: "0dbc15db9893492a8759b2a8e30ce938"
format_stamp: "Formatted at 2026-09-25 22:40:29 on dist-test-slave-wjdr"
server_key: "3082f99b89eca289f373707e8e922511"
server_key_iv: "5aaf227008438ad43d7a3320e20868f3"
server_key_version: "encryptionkey@0"
I20260925 22:40:29.067708 28627 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260925 22:40:29.070159 28642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:29.071483 28627 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 22:40:29.071659 28627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790375752797504-7131-0/minicluster-data/master-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790375752797504-7131-0/minicluster-data/master-0/wal
uuid: "0dbc15db9893492a8759b2a8e30ce938"
format_stamp: "Formatted at 2026-09-25 22:40:29 on dist-test-slave-wjdr"
server_key: "3082f99b89eca289f373707e8e922511"
server_key_iv: "5aaf227008438ad43d7a3320e20868f3"
server_key_version: "encryptionkey@0"
I20260925 22:40:29.071789 28627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790375752797504-7131-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790375752797504-7131-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790375752797504-7131-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:40:29.076678 28627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:40:29.079684 28627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:40:29.079901 28627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:40:29.087289 28627 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38539
I20260925 22:40:29.087281 28694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38539 every 8 connection(s)
I20260925 22:40:29.088227 28627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
I20260925 22:40:29.091018 28695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:40:29.095882 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 28627
I20260925 22:40:29.096019 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790375752797504-7131-0/minicluster-data/master-0/wal/instance
I20260925 22:40:29.096310 7131 external_mini_cluster.cc:1428] Setting key 1aa8d3b1a3c688a3d9595a54a4b80f3b
I20260925 22:40:29.096853 28695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0dbc15db9893492a8759b2a8e30ce938: Bootstrap starting.
I20260925 22:40:29.099328 28695 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0dbc15db9893492a8759b2a8e30ce938: Neither blocks nor log segments found. Creating new log.
I20260925 22:40:29.100303 28695 log.cc:824] T 00000000000000000000000000000000 P 0dbc15db9893492a8759b2a8e30ce938: Log is configured to *not* fsync() on all Append() calls
I20260925 22:40:29.101805 28698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:29.097786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.65:40899 (local address 127.0.0.1:38539)
0925 22:40:29.098302 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:29.098316 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:29.098352 (+ 36us) server_negotiation.cc:407] Connection header received
0925 22:40:29.099002 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:29.099019 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:29.099297 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:29.099620 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:29.100571 (+ 951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:29.100575 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:29.100586 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:40:29.100596 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:29.100650 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:40:29.100669 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:29.100813 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:29.100939 (+ 126us) server_negotiation.cc:299] Negotiation successful
0925 22:40:29.101099 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":186,"threads_started":1}
I20260925 22:40:29.102349 28695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0dbc15db9893492a8759b2a8e30ce938: No bootstrap required, opened a new log
I20260925 22:40:29.104743 7131 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:40:29.106513 28695 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0dbc15db9893492a8759b2a8e30ce938 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dbc15db9893492a8759b2a8e30ce938" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38539 } }
I20260925 22:40:29.106884 28695 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0dbc15db9893492a8759b2a8e30ce938 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:40:29.106962 28695 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0dbc15db9893492a8759b2a8e30ce938 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dbc15db9893492a8759b2a8e30ce938, State: Initialized, Role: FOLLOWER
I20260925 22:40:29.107460 28695 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0dbc15db9893492a8759b2a8e30ce938 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dbc15db9893492a8759b2a8e30ce938" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38539 } }
I20260925 22:40:29.107638 28695 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0dbc15db9893492a8759b2a8e30ce938 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:40:29.107707 28695 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0dbc15db9893492a8759b2a8e30ce938 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:40:29.107792 28695 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0dbc15db9893492a8759b2a8e30ce938 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:40:29.108634 28698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:29.106826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.65:42019 (local address 127.0.0.1:38539)
0925 22:40:29.106952 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:29.106956 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:29.107191 (+ 235us) server_negotiation.cc:407] Connection header received
0925 22:40:29.107290 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:29.107293 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:29.107335 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:29.107401 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:29.108189 (+ 788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:29.108191 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:29.108193 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:40:29.108204 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:29.108228 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:40:29.108238 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:29.108312 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:29.108424 (+ 112us) server_negotiation.cc:299] Negotiation successful
0925 22:40:29.108497 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 22:40:29.109052 28695 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0dbc15db9893492a8759b2a8e30ce938 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dbc15db9893492a8759b2a8e30ce938" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38539 } }
I20260925 22:40:29.109401 28695 leader_election.cc:302] T 00000000000000000000000000000000 P 0dbc15db9893492a8759b2a8e30ce938 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0dbc15db9893492a8759b2a8e30ce938; no voters:
I20260925 22:40:29.109750 28695 leader_election.cc:288] T 00000000000000000000000000000000 P 0dbc15db9893492a8759b2a8e30ce938 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:40:29.109872 28705 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0dbc15db9893492a8759b2a8e30ce938 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:40:29.110103 28705 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0dbc15db9893492a8759b2a8e30ce938 [term 1 LEADER]: Becoming Leader. State: Replica: 0dbc15db9893492a8759b2a8e30ce938, State: Running, Role: LEADER
I20260925 22:40:29.110436 28705 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0dbc15db9893492a8759b2a8e30ce938 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dbc15db9893492a8759b2a8e30ce938" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38539 } }
I20260925 22:40:29.110993 28695 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0dbc15db9893492a8759b2a8e30ce938 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:40:29.112236 28707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0dbc15db9893492a8759b2a8e30ce938 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0dbc15db9893492a8759b2a8e30ce938. Latest consensus state: current_term: 1 leader_uuid: "0dbc15db9893492a8759b2a8e30ce938" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dbc15db9893492a8759b2a8e30ce938" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38539 } } }
I20260925 22:40:29.112296 28706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0dbc15db9893492a8759b2a8e30ce938 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0dbc15db9893492a8759b2a8e30ce938" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dbc15db9893492a8759b2a8e30ce938" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38539 } } }
I20260925 22:40:29.112368 28707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0dbc15db9893492a8759b2a8e30ce938 [sys.catalog]: This master's current role is: LEADER
I20260925 22:40:29.112447 28706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0dbc15db9893492a8759b2a8e30ce938 [sys.catalog]: This master's current role is: LEADER
I20260925 22:40:29.112808 28715 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:40:29.115882 28715 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:40:29.121165 28715 catalog_manager.cc:1347] Generated new cluster ID: 2960d524b3ec4747a0d8e3843d717162
I20260925 22:40:29.121294 28715 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:40:29.131386 28715 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:40:29.131729 28715 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:40:29.140558 28715 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0dbc15db9893492a8759b2a8e30ce938: Generated new TSK 0
I20260925 22:40:29.141322 28715 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:40:29.146464 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 28627
2026-09-25T22:40:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1423 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790375752797504-7131-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:40:29 dist-test-slave-wjdr krb5kdc[28726](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:40:29 dist-test-slave-wjdr krb5kdc[28726](info): set up 2 sockets
Sep 25 22:40:29 dist-test-slave-wjdr krb5kdc[28726](info): commencing operation
krb5kdc: starting...
W20260925 22:40:30.197333 7131 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Sep 25 22:40:30 dist-test-slave-wjdr krb5kdc[28726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:40:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:40:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.246.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab.
Entry for principal kudu/127.6.246.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab.
I20260925 22:40:30.324337 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790375752797504-7131-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790375752797504-7131-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790375752797504-7131-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:42615
--webserver_interface=127.6.246.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab
--principal=kudu/127.6.246.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:34559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.246.254:42615
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:40:30.435997 28739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:40:30.436246 28739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:40:30.436288 28739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:40:30.440268 28739 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:40:30.440331 28739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:40:30.440351 28739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:40:30.440367 28739 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:40:30.440382 28739 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:40:30.445647 28739 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:34559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790375752797504-7131-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790375752797504-7131-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.246.254:42615
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab
--principal=kudu/127.6.246.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:42615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.246.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28739
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790375752797504-7131-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:40:30.446631 28739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:40:30.447568 28739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 22:40:30.456017 28739 server_base.cc:1034] running on GCE node
W20260925 22:40:30.456004 28747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:30.455976 28745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:30.455972 28744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:40:30.456856 28739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:40:30.457893 28739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:40:30.459112 28739 hybrid_clock.cc:648] HybridClock initialized: now 1790376030459085 us; error 44 us; skew 500 ppm
Sep 25 22:40:30 dist-test-slave-wjdr krb5kdc[28726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376030, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:40:30.462034 28739 init.cc:377] Logged in from keytab as kudu/127.6.246.254@KRBTEST.COM (short username kudu)
I20260925 22:40:30.462904 28739 webserver.cc:460] Webserver started at http://127.6.246.254:44315/ using document root <none> and password file <none>
I20260925 22:40:30.463419 28739 fs_manager.cc:359] Metadata directory not provided
I20260925 22:40:30.463478 28739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:40:30.463639 28739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:40:30.465706 28739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790375752797504-7131-0/minicluster-data/master-0/data/instance:
uuid: "23dbf7de0581406eb9a93a7a4bba84fe"
format_stamp: "Formatted at 2026-09-25 22:40:30 on dist-test-slave-wjdr"
server_key: "013acc121223db27bcfaca92082bf7c2"
server_key_iv: "940c562c00fed341494644dff441edaa"
server_key_version: "encryptionkey@0"
I20260925 22:40:30.466202 28739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790375752797504-7131-0/minicluster-data/master-0/wal/instance:
uuid: "23dbf7de0581406eb9a93a7a4bba84fe"
format_stamp: "Formatted at 2026-09-25 22:40:30 on dist-test-slave-wjdr"
server_key: "013acc121223db27bcfaca92082bf7c2"
server_key_iv: "940c562c00fed341494644dff441edaa"
server_key_version: "encryptionkey@0"
I20260925 22:40:30.469887 28739 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:40:30.472421 28754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:30.473673 28739 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 22:40:30.473824 28739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790375752797504-7131-0/minicluster-data/master-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790375752797504-7131-0/minicluster-data/master-0/wal
uuid: "23dbf7de0581406eb9a93a7a4bba84fe"
format_stamp: "Formatted at 2026-09-25 22:40:30 on dist-test-slave-wjdr"
server_key: "013acc121223db27bcfaca92082bf7c2"
server_key_iv: "940c562c00fed341494644dff441edaa"
server_key_version: "encryptionkey@0"
I20260925 22:40:30.473934 28739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790375752797504-7131-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790375752797504-7131-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790375752797504-7131-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:40:30.492547 28739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:40:30.495507 28739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:40:30.495730 28739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:40:30.502971 28739 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.254:42615
I20260925 22:40:30.502990 28806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.254:42615 every 8 connection(s)
I20260925 22:40:30.503904 28739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
I20260925 22:40:30.506562 28807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:40:30.511399 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 28739
I20260925 22:40:30.511572 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790375752797504-7131-0/minicluster-data/master-0/wal/instance
I20260925 22:40:30.511849 7131 external_mini_cluster.cc:1428] Setting key 2b10e6383809f10d96d0e0b82201dde8
I20260925 22:40:30.512339 28807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23dbf7de0581406eb9a93a7a4bba84fe: Bootstrap starting.
I20260925 22:40:30.514993 28807 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23dbf7de0581406eb9a93a7a4bba84fe: Neither blocks nor log segments found. Creating new log.
I20260925 22:40:30.515864 28807 log.cc:824] T 00000000000000000000000000000000 P 23dbf7de0581406eb9a93a7a4bba84fe: Log is configured to *not* fsync() on all Append() calls
I20260925 22:40:30.518267 28807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23dbf7de0581406eb9a93a7a4bba84fe: No bootstrap required, opened a new log
Sep 25 22:40:30 dist-test-slave-wjdr krb5kdc[28726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
I20260925 22:40:30.522274 28807 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23dbf7de0581406eb9a93a7a4bba84fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23dbf7de0581406eb9a93a7a4bba84fe" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 42615 } }
I20260925 22:40:30.522583 28807 raft_consensus.cc:374] T 00000000000000000000000000000000 P 23dbf7de0581406eb9a93a7a4bba84fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:40:30.522665 28807 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23dbf7de0581406eb9a93a7a4bba84fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23dbf7de0581406eb9a93a7a4bba84fe, State: Initialized, Role: FOLLOWER
I20260925 22:40:30.523198 28807 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23dbf7de0581406eb9a93a7a4bba84fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23dbf7de0581406eb9a93a7a4bba84fe" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 42615 } }
I20260925 22:40:30.523396 28807 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23dbf7de0581406eb9a93a7a4bba84fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:40:30.523476 28807 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23dbf7de0581406eb9a93a7a4bba84fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:40:30.523592 28807 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23dbf7de0581406eb9a93a7a4bba84fe [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:40:30.524762 28807 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23dbf7de0581406eb9a93a7a4bba84fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23dbf7de0581406eb9a93a7a4bba84fe" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 42615 } }
I20260925 22:40:30.525125 28807 leader_election.cc:302] T 00000000000000000000000000000000 P 23dbf7de0581406eb9a93a7a4bba84fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23dbf7de0581406eb9a93a7a4bba84fe; no voters:
I20260925 22:40:30.525494 28807 leader_election.cc:288] T 00000000000000000000000000000000 P 23dbf7de0581406eb9a93a7a4bba84fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:40:30.525637 28812 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23dbf7de0581406eb9a93a7a4bba84fe [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:40:30.525915 28812 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23dbf7de0581406eb9a93a7a4bba84fe [term 1 LEADER]: Becoming Leader. State: Replica: 23dbf7de0581406eb9a93a7a4bba84fe, State: Running, Role: LEADER
I20260925 22:40:30.526257 28812 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23dbf7de0581406eb9a93a7a4bba84fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23dbf7de0581406eb9a93a7a4bba84fe" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 42615 } }
I20260925 22:40:30.526809 28807 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23dbf7de0581406eb9a93a7a4bba84fe [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:40:30.527789 28814 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23dbf7de0581406eb9a93a7a4bba84fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23dbf7de0581406eb9a93a7a4bba84fe. Latest consensus state: current_term: 1 leader_uuid: "23dbf7de0581406eb9a93a7a4bba84fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23dbf7de0581406eb9a93a7a4bba84fe" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 42615 } } }
I20260925 22:40:30.527911 28814 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23dbf7de0581406eb9a93a7a4bba84fe [sys.catalog]: This master's current role is: LEADER
I20260925 22:40:30.528249 28813 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23dbf7de0581406eb9a93a7a4bba84fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "23dbf7de0581406eb9a93a7a4bba84fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23dbf7de0581406eb9a93a7a4bba84fe" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 42615 } } }
I20260925 22:40:30.528297 28810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:30.513323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49315 (local address 127.6.246.254:42615)
0925 22:40:30.513769 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:30.513789 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:30.513832 (+ 43us) server_negotiation.cc:407] Connection header received
0925 22:40:30.514658 (+ 826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:30.514679 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:30.515109 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:30.515476 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:30.516238 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:30.517799 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:30.518722 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:30.519404 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:30.519560 (+ 156us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:30.522161 (+ 2601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:30.522168 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:30.522184 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:30.522212 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:30.524339 (+ 2127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:30.524927 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:30.524933 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:30.524939 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:30.525069 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:30.525426 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:30.525432 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:30.525437 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:30.525975 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:30.526853 (+ 878us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:30.527026 (+ 173us) server_negotiation.cc:299] Negotiation successful
0925 22:40:30.527248 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":287,"thread_start_us":123,"threads_started":1}
I20260925 22:40:30.528326 28813 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23dbf7de0581406eb9a93a7a4bba84fe [sys.catalog]: This master's current role is: LEADER
I20260925 22:40:30.531328 7131 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260925 22:40:30.531750 28827 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 23dbf7de0581406eb9a93a7a4bba84fe: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:40:30.531852 28827 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:40:30.531960 28821 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:40:30.532965 28821 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:40:30.539361 28821 catalog_manager.cc:1347] Generated new cluster ID: 88370d9584a1446197e7af22a888ac56
I20260925 22:40:30.539484 28821 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:40:30.544679 28810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:30.533585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46075 (local address 127.6.246.254:42615)
0925 22:40:30.533692 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:30.533696 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:30.533992 (+ 296us) server_negotiation.cc:407] Connection header received
0925 22:40:30.534106 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:30.534109 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:30.534156 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:30.534246 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:30.536578 (+ 2332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:30.537124 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:30.538300 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:30.538631 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:30.538741 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:30.541403 (+ 2662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:30.541406 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:30.541409 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:30.541430 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:30.542875 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:30.543356 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:30.543361 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:30.543363 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:30.543439 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:30.543770 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:30.543775 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:30.543777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:30.543950 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:30.544107 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:30.544373 (+ 266us) server_negotiation.cc:299] Negotiation successful
0925 22:40:30.544493 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 22:40:30.549036 28821 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:40:30.549984 28821 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:40:30.558007 28821 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 23dbf7de0581406eb9a93a7a4bba84fe: Generated new TSK 0
I20260925 22:40:30.558797 28821 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:40:30.562284 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 28739
W20260925 22:40:30.567058 28724 connection.cc:535] client connection to 127.6.246.254:42615 recv error: Network error: failed to read from TLS socket (remote: 127.6.246.254:42615): Connection reset by peer (error 104)
2026-09-25T22:40:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1414 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790375752797504-7131-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:40:30 dist-test-slave-wjdr krb5kdc[28838](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:40:30 dist-test-slave-wjdr krb5kdc[28838](info): set up 2 sockets
Sep 25 22:40:30 dist-test-slave-wjdr krb5kdc[28838](info): commencing operation
krb5kdc: starting...
W20260925 22:40:31.604820 7131 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Sep 25 22:40:31 dist-test-slave-wjdr krb5kdc[28838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:40:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:40:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.246.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab.
Entry for principal kudu/127.6.246.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab.
I20260925 22:40:31.725538 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790375752797504-7131-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790375752797504-7131-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790375752797504-7131-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:40609
--webserver_interface=127.6.246.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab
--principal=kudu/127.6.246.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:34091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.246.254:40609
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:40:31.837430 28851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:40:31.837755 28851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:40:31.837860 28851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:40:31.841931 28851 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:40:31.842013 28851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:40:31.842051 28851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:40:31.842108 28851 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:40:31.842142 28851 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:40:31.847335 28851 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:34091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790375752797504-7131-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790375752797504-7131-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.246.254:40609
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab
--principal=kudu/127.6.246.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:40609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.246.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28851
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790375752797504-7131-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:40:31.848371 28851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:40:31.849376 28851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:40:31.857810 28857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:31.857823 28859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:31.857822 28856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:40:31.858019 28851 server_base.cc:1034] running on GCE node
I20260925 22:40:31.858810 28851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:40:31.859872 28851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:40:31.861033 28851 hybrid_clock.cc:648] HybridClock initialized: now 1790376031861007 us; error 38 us; skew 500 ppm
Sep 25 22:40:31 dist-test-slave-wjdr krb5kdc[28838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376031, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:40:31.866786 28851 init.cc:377] Logged in from keytab as kudu/127.6.246.254@KRBTEST.COM (short username kudu)
I20260925 22:40:31.867643 28851 webserver.cc:460] Webserver started at http://127.6.246.254:39853/ using document root <none> and password file <none>
I20260925 22:40:31.868198 28851 fs_manager.cc:359] Metadata directory not provided
I20260925 22:40:31.868257 28851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:40:31.868553 28851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:40:31.870509 28851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790375752797504-7131-0/minicluster-data/master-0/data/instance:
uuid: "c39bc97c902c4fb59258fc3784c88376"
format_stamp: "Formatted at 2026-09-25 22:40:31 on dist-test-slave-wjdr"
server_key: "42ca9ef4a33422c2f08d876e57ecd663"
server_key_iv: "9dc7aa2ff5043129d9f399ca7ad0dfdc"
server_key_version: "encryptionkey@0"
I20260925 22:40:31.871057 28851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790375752797504-7131-0/minicluster-data/master-0/wal/instance:
uuid: "c39bc97c902c4fb59258fc3784c88376"
format_stamp: "Formatted at 2026-09-25 22:40:31 on dist-test-slave-wjdr"
server_key: "42ca9ef4a33422c2f08d876e57ecd663"
server_key_iv: "9dc7aa2ff5043129d9f399ca7ad0dfdc"
server_key_version: "encryptionkey@0"
I20260925 22:40:31.874656 28851 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 22:40:31.876968 28866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:31.878132 28851 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:40:31.878278 28851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790375752797504-7131-0/minicluster-data/master-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790375752797504-7131-0/minicluster-data/master-0/wal
uuid: "c39bc97c902c4fb59258fc3784c88376"
format_stamp: "Formatted at 2026-09-25 22:40:31 on dist-test-slave-wjdr"
server_key: "42ca9ef4a33422c2f08d876e57ecd663"
server_key_iv: "9dc7aa2ff5043129d9f399ca7ad0dfdc"
server_key_version: "encryptionkey@0"
I20260925 22:40:31.878383 28851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790375752797504-7131-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790375752797504-7131-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790375752797504-7131-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:40:31.896541 28851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:40:31.899531 28851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:40:31.899739 28851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:40:31.906788 28851 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.254:40609
I20260925 22:40:31.906816 28918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.254:40609 every 8 connection(s)
I20260925 22:40:31.907687 28851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
I20260925 22:40:31.910332 28919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:40:31.911697 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 28851
I20260925 22:40:31.911842 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790375752797504-7131-0/minicluster-data/master-0/wal/instance
I20260925 22:40:31.912163 7131 external_mini_cluster.cc:1428] Setting key 68e0b4de891e08e8daa7ad447dc6fc49
I20260925 22:40:31.916343 28919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c39bc97c902c4fb59258fc3784c88376: Bootstrap starting.
I20260925 22:40:31.918510 28919 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c39bc97c902c4fb59258fc3784c88376: Neither blocks nor log segments found. Creating new log.
I20260925 22:40:31.919322 28919 log.cc:824] T 00000000000000000000000000000000 P c39bc97c902c4fb59258fc3784c88376: Log is configured to *not* fsync() on all Append() calls
I20260925 22:40:31.921130 28922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:31.913622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36005 (local address 127.6.246.254:40609)
0925 22:40:31.914051 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:31.914064 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:31.914097 (+ 33us) server_negotiation.cc:407] Connection header received
0925 22:40:31.914713 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:31.914728 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:31.915040 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:31.915366 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:31.915759 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:31.916775 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:31.917651 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:31.918110 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:31.918285 (+ 175us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:31.919188 (+ 903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:31.919193 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:31.919205 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:40:31.919637 (+ 432us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:31.919688 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:40:31.919711 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:31.919847 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:31.920047 (+ 200us) server_negotiation.cc:299] Negotiation successful
0925 22:40:31.920178 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":119,"threads_started":1}
I20260925 22:40:31.921317 28919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c39bc97c902c4fb59258fc3784c88376: No bootstrap required, opened a new log
I20260925 22:40:31.924078 7131 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:40:31.925470 28919 raft_consensus.cc:348] T 00000000000000000000000000000000 P c39bc97c902c4fb59258fc3784c88376 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c39bc97c902c4fb59258fc3784c88376" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 40609 } }
I20260925 22:40:31.925709 28919 raft_consensus.cc:374] T 00000000000000000000000000000000 P c39bc97c902c4fb59258fc3784c88376 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:40:31.925791 28919 raft_consensus.cc:729] T 00000000000000000000000000000000 P c39bc97c902c4fb59258fc3784c88376 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c39bc97c902c4fb59258fc3784c88376, State: Initialized, Role: FOLLOWER
I20260925 22:40:31.926254 28919 consensus_queue.cc:260] T 00000000000000000000000000000000 P c39bc97c902c4fb59258fc3784c88376 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c39bc97c902c4fb59258fc3784c88376" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 40609 } }
I20260925 22:40:31.926410 28919 raft_consensus.cc:388] T 00000000000000000000000000000000 P c39bc97c902c4fb59258fc3784c88376 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:40:31.926481 28919 raft_consensus.cc:482] T 00000000000000000000000000000000 P c39bc97c902c4fb59258fc3784c88376 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:40:31.926584 28919 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c39bc97c902c4fb59258fc3784c88376 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:40:31.927839 28919 raft_consensus.cc:504] T 00000000000000000000000000000000 P c39bc97c902c4fb59258fc3784c88376 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c39bc97c902c4fb59258fc3784c88376" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 40609 } }
I20260925 22:40:31.928195 28919 leader_election.cc:302] T 00000000000000000000000000000000 P c39bc97c902c4fb59258fc3784c88376 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c39bc97c902c4fb59258fc3784c88376; no voters:
I20260925 22:40:31.928532 28919 leader_election.cc:288] T 00000000000000000000000000000000 P c39bc97c902c4fb59258fc3784c88376 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:40:31.928617 28929 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c39bc97c902c4fb59258fc3784c88376 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:40:31.928900 28929 raft_consensus.cc:686] T 00000000000000000000000000000000 P c39bc97c902c4fb59258fc3784c88376 [term 1 LEADER]: Becoming Leader. State: Replica: c39bc97c902c4fb59258fc3784c88376, State: Running, Role: LEADER
I20260925 22:40:31.929222 28929 consensus_queue.cc:237] T 00000000000000000000000000000000 P c39bc97c902c4fb59258fc3784c88376 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c39bc97c902c4fb59258fc3784c88376" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 40609 } }
I20260925 22:40:31.929908 28919 sys_catalog.cc:564] T 00000000000000000000000000000000 P c39bc97c902c4fb59258fc3784c88376 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:40:31.930922 28922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:31.926047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33407 (local address 127.6.246.254:40609)
0925 22:40:31.926171 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:31.926175 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:31.926404 (+ 229us) server_negotiation.cc:407] Connection header received
0925 22:40:31.926548 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:31.926552 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:31.926604 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:31.926697 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:31.926959 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:31.927738 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:31.928736 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:31.929069 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:31.929169 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:31.930351 (+ 1182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:31.930355 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:31.930358 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:40:31.930369 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:31.930396 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:40:31.930409 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:31.930496 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:31.930712 (+ 216us) server_negotiation.cc:299] Negotiation successful
0925 22:40:31.930752 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 22:40:31.930867 28931 sys_catalog.cc:455] T 00000000000000000000000000000000 P c39bc97c902c4fb59258fc3784c88376 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c39bc97c902c4fb59258fc3784c88376. Latest consensus state: current_term: 1 leader_uuid: "c39bc97c902c4fb59258fc3784c88376" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c39bc97c902c4fb59258fc3784c88376" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 40609 } } }
I20260925 22:40:31.931005 28931 sys_catalog.cc:458] T 00000000000000000000000000000000 P c39bc97c902c4fb59258fc3784c88376 [sys.catalog]: This master's current role is: LEADER
I20260925 22:40:31.930980 28930 sys_catalog.cc:455] T 00000000000000000000000000000000 P c39bc97c902c4fb59258fc3784c88376 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c39bc97c902c4fb59258fc3784c88376" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c39bc97c902c4fb59258fc3784c88376" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 40609 } } }
I20260925 22:40:31.931046 28930 sys_catalog.cc:458] T 00000000000000000000000000000000 P c39bc97c902c4fb59258fc3784c88376 [sys.catalog]: This master's current role is: LEADER
W20260925 22:40:31.934487 28944 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c39bc97c902c4fb59258fc3784c88376: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:40:31.934563 28944 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:40:31.934645 28938 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:40:31.935341 28938 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:40:31.940779 28938 catalog_manager.cc:1347] Generated new cluster ID: 3f6c16fe4c58402b916db4159d31db09
I20260925 22:40:31.940848 28938 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:40:31.948558 28938 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:40:31.949694 28938 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:40:31.954386 28938 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c39bc97c902c4fb59258fc3784c88376: Generated new TSK 0
I20260925 22:40:31.955147 28938 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:40:31.967294 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 28851
W20260925 22:40:31.972914 28836 connection.cc:535] client connection to 127.6.246.254:40609 recv error: Network error: failed to read from TLS socket (remote: 127.6.246.254:40609): Connection reset by peer (error 104)
2026-09-25T22:40:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1405 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790375752797504-7131-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:40:32 dist-test-slave-wjdr krb5kdc[28950](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:40:32 dist-test-slave-wjdr krb5kdc[28950](info): set up 2 sockets
Sep 25 22:40:32 dist-test-slave-wjdr krb5kdc[28950](info): commencing operation
krb5kdc: starting...
W20260925 22:40:33.004890 7131 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Sep 25 22:40:33 dist-test-slave-wjdr krb5kdc[28950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:40:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:40:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.246.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab.
Entry for principal kudu/127.6.246.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab.
I20260925 22:40:33.128437 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790375752797504-7131-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790375752797504-7131-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790375752797504-7131-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:46449
--webserver_interface=127.6.246.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab
--principal=kudu/127.6.246.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:40787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.246.254:46449
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:40:33.238046 28963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:40:33.238344 28963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:40:33.238422 28963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:40:33.242439 28963 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:40:33.242537 28963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:40:33.242575 28963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:40:33.242633 28963 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:40:33.242671 28963 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:40:33.247939 28963 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:40787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790375752797504-7131-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790375752797504-7131-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.246.254:46449
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab
--principal=kudu/127.6.246.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:46449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.246.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28963
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790375752797504-7131-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:40:33.249034 28963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:40:33.250046 28963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:40:33.258112 28971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:33.258140 28969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:33.258112 28968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:40:33.258553 28963 server_base.cc:1034] running on GCE node
I20260925 22:40:33.259083 28963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:40:33.260138 28963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:40:33.261304 28963 hybrid_clock.cc:648] HybridClock initialized: now 1790376033261291 us; error 25 us; skew 500 ppm
Sep 25 22:40:33 dist-test-slave-wjdr krb5kdc[28950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376033, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:40:33.264071 28963 init.cc:377] Logged in from keytab as kudu/127.6.246.254@KRBTEST.COM (short username kudu)
I20260925 22:40:33.264973 28963 webserver.cc:460] Webserver started at http://127.6.246.254:45699/ using document root <none> and password file <none>
I20260925 22:40:33.265522 28963 fs_manager.cc:359] Metadata directory not provided
I20260925 22:40:33.265614 28963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:40:33.265848 28963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:40:33.267797 28963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790375752797504-7131-0/minicluster-data/master-0/data/instance:
uuid: "725edef14ebc4df7b46d058e14a7a8a9"
format_stamp: "Formatted at 2026-09-25 22:40:33 on dist-test-slave-wjdr"
server_key: "6720c60575844a483592d83cddac6b50"
server_key_iv: "0e470f2280daea016f3cf2d8da26e1d6"
server_key_version: "encryptionkey@0"
I20260925 22:40:33.268358 28963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790375752797504-7131-0/minicluster-data/master-0/wal/instance:
uuid: "725edef14ebc4df7b46d058e14a7a8a9"
format_stamp: "Formatted at 2026-09-25 22:40:33 on dist-test-slave-wjdr"
server_key: "6720c60575844a483592d83cddac6b50"
server_key_iv: "0e470f2280daea016f3cf2d8da26e1d6"
server_key_version: "encryptionkey@0"
I20260925 22:40:33.271862 28963 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:40:33.274209 28978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:33.275292 28963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 22:40:33.275437 28963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790375752797504-7131-0/minicluster-data/master-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790375752797504-7131-0/minicluster-data/master-0/wal
uuid: "725edef14ebc4df7b46d058e14a7a8a9"
format_stamp: "Formatted at 2026-09-25 22:40:33 on dist-test-slave-wjdr"
server_key: "6720c60575844a483592d83cddac6b50"
server_key_iv: "0e470f2280daea016f3cf2d8da26e1d6"
server_key_version: "encryptionkey@0"
I20260925 22:40:33.275544 28963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790375752797504-7131-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790375752797504-7131-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790375752797504-7131-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:40:33.279986 28963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:40:33.282764 28963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:40:33.282913 28963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:40:33.289887 28963 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.254:46449
I20260925 22:40:33.289917 29030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.254:46449 every 8 connection(s)
I20260925 22:40:33.290824 28963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
I20260925 22:40:33.293548 29031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:40:33.294656 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 28963
I20260925 22:40:33.294791 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790375752797504-7131-0/minicluster-data/master-0/wal/instance
I20260925 22:40:33.295106 7131 external_mini_cluster.cc:1428] Setting key 4d0aec2f5fae60621fb8f216f786417a
I20260925 22:40:33.299782 29031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 725edef14ebc4df7b46d058e14a7a8a9: Bootstrap starting.
I20260925 22:40:33.300189 29034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:33.296500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36177 (local address 127.6.246.254:46449)
0925 22:40:33.296960 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:33.296973 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:33.297007 (+ 34us) server_negotiation.cc:407] Connection header received
0925 22:40:33.297604 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:33.297618 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:33.297888 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:33.298208 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:33.298992 (+ 784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:33.298998 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:33.299013 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:40:33.299031 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:33.299083 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:40:33.299101 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:33.299252 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:33.299321 (+ 69us) server_negotiation.cc:299] Negotiation successful
0925 22:40:33.299470 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":171,"threads_started":1}
I20260925 22:40:33.302237 29031 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 725edef14ebc4df7b46d058e14a7a8a9: Neither blocks nor log segments found. Creating new log.
I20260925 22:40:33.302870 7131 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:40:33.302958 29031 log.cc:824] T 00000000000000000000000000000000 P 725edef14ebc4df7b46d058e14a7a8a9: Log is configured to *not* fsync() on all Append() calls
I20260925 22:40:33.305007 29031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 725edef14ebc4df7b46d058e14a7a8a9: No bootstrap required, opened a new log
I20260925 22:40:33.306435 29034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:33.304832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59423 (local address 127.6.246.254:46449)
0925 22:40:33.304944 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:33.304947 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:33.305178 (+ 231us) server_negotiation.cc:407] Connection header received
0925 22:40:33.305295 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:33.305297 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:33.305335 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:33.305398 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:33.306051 (+ 653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:33.306054 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:33.306056 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:40:33.306065 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:33.306087 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:40:33.306096 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:33.306161 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:33.306262 (+ 101us) server_negotiation.cc:299] Negotiation successful
0925 22:40:33.306291 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 22:40:33.309146 29031 raft_consensus.cc:348] T 00000000000000000000000000000000 P 725edef14ebc4df7b46d058e14a7a8a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "725edef14ebc4df7b46d058e14a7a8a9" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 46449 } }
I20260925 22:40:33.309361 29031 raft_consensus.cc:374] T 00000000000000000000000000000000 P 725edef14ebc4df7b46d058e14a7a8a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:40:33.309437 29031 raft_consensus.cc:729] T 00000000000000000000000000000000 P 725edef14ebc4df7b46d058e14a7a8a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 725edef14ebc4df7b46d058e14a7a8a9, State: Initialized, Role: FOLLOWER
I20260925 22:40:33.309876 29031 consensus_queue.cc:260] T 00000000000000000000000000000000 P 725edef14ebc4df7b46d058e14a7a8a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "725edef14ebc4df7b46d058e14a7a8a9" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 46449 } }
I20260925 22:40:33.310026 29031 raft_consensus.cc:388] T 00000000000000000000000000000000 P 725edef14ebc4df7b46d058e14a7a8a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:40:33.310110 29031 raft_consensus.cc:482] T 00000000000000000000000000000000 P 725edef14ebc4df7b46d058e14a7a8a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:40:33.310209 29031 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 725edef14ebc4df7b46d058e14a7a8a9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:40:33.311259 29031 raft_consensus.cc:504] T 00000000000000000000000000000000 P 725edef14ebc4df7b46d058e14a7a8a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "725edef14ebc4df7b46d058e14a7a8a9" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 46449 } }
I20260925 22:40:33.311589 29031 leader_election.cc:302] T 00000000000000000000000000000000 P 725edef14ebc4df7b46d058e14a7a8a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 725edef14ebc4df7b46d058e14a7a8a9; no voters:
I20260925 22:40:33.311909 29031 leader_election.cc:288] T 00000000000000000000000000000000 P 725edef14ebc4df7b46d058e14a7a8a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:40:33.312007 29041 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 725edef14ebc4df7b46d058e14a7a8a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:40:33.312256 29041 raft_consensus.cc:686] T 00000000000000000000000000000000 P 725edef14ebc4df7b46d058e14a7a8a9 [term 1 LEADER]: Becoming Leader. State: Replica: 725edef14ebc4df7b46d058e14a7a8a9, State: Running, Role: LEADER
I20260925 22:40:33.312652 29041 consensus_queue.cc:237] T 00000000000000000000000000000000 P 725edef14ebc4df7b46d058e14a7a8a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "725edef14ebc4df7b46d058e14a7a8a9" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 46449 } }
I20260925 22:40:33.313112 29031 sys_catalog.cc:564] T 00000000000000000000000000000000 P 725edef14ebc4df7b46d058e14a7a8a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:40:33.314458 29042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 725edef14ebc4df7b46d058e14a7a8a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "725edef14ebc4df7b46d058e14a7a8a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "725edef14ebc4df7b46d058e14a7a8a9" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 46449 } } }
I20260925 22:40:33.314488 29043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 725edef14ebc4df7b46d058e14a7a8a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 725edef14ebc4df7b46d058e14a7a8a9. Latest consensus state: current_term: 1 leader_uuid: "725edef14ebc4df7b46d058e14a7a8a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "725edef14ebc4df7b46d058e14a7a8a9" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 46449 } } }
I20260925 22:40:33.314651 29043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 725edef14ebc4df7b46d058e14a7a8a9 [sys.catalog]: This master's current role is: LEADER
I20260925 22:40:33.314644 29042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 725edef14ebc4df7b46d058e14a7a8a9 [sys.catalog]: This master's current role is: LEADER
I20260925 22:40:33.315196 29052 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:40:33.317943 29052 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:40:33.322917 29052 catalog_manager.cc:1347] Generated new cluster ID: 66dc16ba08054015a8b427a203d1be7f
I20260925 22:40:33.322979 29052 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:40:33.337108 29052 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:40:33.337422 29052 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:40:33.342260 29052 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 725edef14ebc4df7b46d058e14a7a8a9: Generated new TSK 0
I20260925 22:40:33.342904 29052 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:40:33.348348 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 28963
2026-09-25T22:40:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1380 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790375752797504-7131-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:40:33 dist-test-slave-wjdr krb5kdc[29062](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:40:33 dist-test-slave-wjdr krb5kdc[29062](info): set up 2 sockets
Sep 25 22:40:33 dist-test-slave-wjdr krb5kdc[29062](info): commencing operation
krb5kdc: starting...
W20260925 22:40:34.372758 7131 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Sep 25 22:40:34 dist-test-slave-wjdr krb5kdc[29062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376034, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:40:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:40:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.246.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab.
Entry for principal kudu/127.6.246.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab.
I20260925 22:40:34.491070 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790375752797504-7131-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790375752797504-7131-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790375752797504-7131-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:36963
--webserver_interface=127.6.246.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab
--principal=kudu/127.6.246.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:36935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.246.254:36963
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:40:34.599344 29075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:40:34.599664 29075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:40:34.599743 29075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:40:34.603715 29075 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:40:34.603807 29075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:40:34.603848 29075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:40:34.603888 29075 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:40:34.603925 29075 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:40:34.609217 29075 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:36935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790375752797504-7131-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790375752797504-7131-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.246.254:36963
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab
--principal=kudu/127.6.246.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:36963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.246.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29075
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790375752797504-7131-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:40:34.610291 29075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:40:34.611294 29075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:40:34.619834 29080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:34.619966 29081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:34.619834 29083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:40:34.619903 29075 server_base.cc:1034] running on GCE node
I20260925 22:40:34.620743 29075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:40:34.621783 29075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:40:34.622941 29075 hybrid_clock.cc:648] HybridClock initialized: now 1790376034622916 us; error 44 us; skew 500 ppm
Sep 25 22:40:34 dist-test-slave-wjdr krb5kdc[29062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376034, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:40:34.625828 29075 init.cc:377] Logged in from keytab as kudu/127.6.246.254@KRBTEST.COM (short username kudu)
I20260925 22:40:34.626752 29075 webserver.cc:460] Webserver started at http://127.6.246.254:44601/ using document root <none> and password file <none>
I20260925 22:40:34.627310 29075 fs_manager.cc:359] Metadata directory not provided
I20260925 22:40:34.627398 29075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:40:34.627624 29075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:40:34.629735 29075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790375752797504-7131-0/minicluster-data/master-0/data/instance:
uuid: "190388cc6f5245928266230716e0dd83"
format_stamp: "Formatted at 2026-09-25 22:40:34 on dist-test-slave-wjdr"
server_key: "aee880452c3c6987d519902a84c12247"
server_key_iv: "4393b85e230fbd4d98eda39dac54a654"
server_key_version: "encryptionkey@0"
I20260925 22:40:34.630293 29075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790375752797504-7131-0/minicluster-data/master-0/wal/instance:
uuid: "190388cc6f5245928266230716e0dd83"
format_stamp: "Formatted at 2026-09-25 22:40:34 on dist-test-slave-wjdr"
server_key: "aee880452c3c6987d519902a84c12247"
server_key_iv: "4393b85e230fbd4d98eda39dac54a654"
server_key_version: "encryptionkey@0"
I20260925 22:40:34.633932 29075 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:40:34.636283 29090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:34.637488 29075 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:40:34.637627 29075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790375752797504-7131-0/minicluster-data/master-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790375752797504-7131-0/minicluster-data/master-0/wal
uuid: "190388cc6f5245928266230716e0dd83"
format_stamp: "Formatted at 2026-09-25 22:40:34 on dist-test-slave-wjdr"
server_key: "aee880452c3c6987d519902a84c12247"
server_key_iv: "4393b85e230fbd4d98eda39dac54a654"
server_key_version: "encryptionkey@0"
I20260925 22:40:34.637738 29075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790375752797504-7131-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790375752797504-7131-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790375752797504-7131-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:40:34.650900 29075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:40:34.653594 29075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:40:34.653818 29075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:40:34.660569 29075 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.254:36963
I20260925 22:40:34.660593 29142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.254:36963 every 8 connection(s)
I20260925 22:40:34.661458 29075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
I20260925 22:40:34.664012 29143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:40:34.666858 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 29075
I20260925 22:40:34.666998 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790375752797504-7131-0/minicluster-data/master-0/wal/instance
I20260925 22:40:34.667279 7131 external_mini_cluster.cc:1428] Setting key 84c2aa6f061643adff33ba00aeeb086d
I20260925 22:40:34.670029 29143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 190388cc6f5245928266230716e0dd83: Bootstrap starting.
I20260925 22:40:34.672322 29143 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 190388cc6f5245928266230716e0dd83: Neither blocks nor log segments found. Creating new log.
I20260925 22:40:34.673055 29143 log.cc:824] T 00000000000000000000000000000000 P 190388cc6f5245928266230716e0dd83: Log is configured to *not* fsync() on all Append() calls
I20260925 22:40:34.674988 29143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 190388cc6f5245928266230716e0dd83: No bootstrap required, opened a new log
Sep 25 22:40:34 dist-test-slave-wjdr krb5kdc[29062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376034, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
I20260925 22:40:34.678340 29143 raft_consensus.cc:348] T 00000000000000000000000000000000 P 190388cc6f5245928266230716e0dd83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "190388cc6f5245928266230716e0dd83" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 36963 } }
I20260925 22:40:34.678541 29143 raft_consensus.cc:374] T 00000000000000000000000000000000 P 190388cc6f5245928266230716e0dd83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:40:34.678608 29143 raft_consensus.cc:729] T 00000000000000000000000000000000 P 190388cc6f5245928266230716e0dd83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 190388cc6f5245928266230716e0dd83, State: Initialized, Role: FOLLOWER
I20260925 22:40:34.679072 29143 consensus_queue.cc:260] T 00000000000000000000000000000000 P 190388cc6f5245928266230716e0dd83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "190388cc6f5245928266230716e0dd83" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 36963 } }
I20260925 22:40:34.679214 29143 raft_consensus.cc:388] T 00000000000000000000000000000000 P 190388cc6f5245928266230716e0dd83 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:40:34.679286 29143 raft_consensus.cc:482] T 00000000000000000000000000000000 P 190388cc6f5245928266230716e0dd83 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:40:34.679401 29143 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 190388cc6f5245928266230716e0dd83 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:40:34.680210 29143 raft_consensus.cc:504] T 00000000000000000000000000000000 P 190388cc6f5245928266230716e0dd83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "190388cc6f5245928266230716e0dd83" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 36963 } }
I20260925 22:40:34.680595 29143 leader_election.cc:302] T 00000000000000000000000000000000 P 190388cc6f5245928266230716e0dd83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 190388cc6f5245928266230716e0dd83; no voters:
I20260925 22:40:34.680922 29143 leader_election.cc:288] T 00000000000000000000000000000000 P 190388cc6f5245928266230716e0dd83 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:40:34.681010 29148 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 190388cc6f5245928266230716e0dd83 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:40:34.681255 29148 raft_consensus.cc:686] T 00000000000000000000000000000000 P 190388cc6f5245928266230716e0dd83 [term 1 LEADER]: Becoming Leader. State: Replica: 190388cc6f5245928266230716e0dd83, State: Running, Role: LEADER
I20260925 22:40:34.681607 29148 consensus_queue.cc:237] T 00000000000000000000000000000000 P 190388cc6f5245928266230716e0dd83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "190388cc6f5245928266230716e0dd83" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 36963 } }
I20260925 22:40:34.682003 29146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:34.668721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58557 (local address 127.6.246.254:36963)
0925 22:40:34.669151 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:34.669166 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:34.669198 (+ 32us) server_negotiation.cc:407] Connection header received
0925 22:40:34.669780 (+ 582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:34.669799 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:34.670139 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:34.670466 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:34.671263 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:34.672299 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:34.673190 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:34.673868 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:34.674049 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:34.676753 (+ 2704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:34.676758 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:34.676770 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:34.676795 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:34.678992 (+ 2197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:34.679473 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:34.679478 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:34.679482 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:34.679621 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:34.679911 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:34.679914 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:34.679916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:34.680270 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:34.680515 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:34.680750 (+ 235us) server_negotiation.cc:299] Negotiation successful
0925 22:40:34.681079 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":135,"threads_started":1}
I20260925 22:40:34.682206 29143 sys_catalog.cc:564] T 00000000000000000000000000000000 P 190388cc6f5245928266230716e0dd83 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:40:34.685151 29150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 190388cc6f5245928266230716e0dd83 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 190388cc6f5245928266230716e0dd83. Latest consensus state: current_term: 1 leader_uuid: "190388cc6f5245928266230716e0dd83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "190388cc6f5245928266230716e0dd83" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 36963 } } }
I20260925 22:40:34.684686 29149 sys_catalog.cc:455] T 00000000000000000000000000000000 P 190388cc6f5245928266230716e0dd83 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "190388cc6f5245928266230716e0dd83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "190388cc6f5245928266230716e0dd83" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 36963 } } }
I20260925 22:40:34.685343 29149 sys_catalog.cc:458] T 00000000000000000000000000000000 P 190388cc6f5245928266230716e0dd83 [sys.catalog]: This master's current role is: LEADER
I20260925 22:40:34.685578 29150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 190388cc6f5245928266230716e0dd83 [sys.catalog]: This master's current role is: LEADER
I20260925 22:40:34.685912 7131 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260925 22:40:34.687070 29163 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 190388cc6f5245928266230716e0dd83: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:40:34.687201 29163 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:40:34.687291 29164 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:40:34.688167 29164 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:40:34.693285 29164 catalog_manager.cc:1347] Generated new cluster ID: 60815f4705274a939a9a220f05a6e338
I20260925 22:40:34.693338 29164 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:40:34.697243 29146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:34.688082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43403 (local address 127.6.246.254:36963)
0925 22:40:34.688180 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:34.688186 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:34.688518 (+ 332us) server_negotiation.cc:407] Connection header received
0925 22:40:34.688658 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:34.688661 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:34.688705 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:34.688787 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:34.689574 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:34.690332 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:34.691587 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:34.691955 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:34.692035 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:34.693861 (+ 1826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:34.693865 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:34.693868 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:34.693891 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:34.695462 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:34.695941 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:34.695944 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:34.695945 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:34.696021 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:34.696307 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:34.696310 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:34.696311 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:34.696513 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:34.696625 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:34.696877 (+ 252us) server_negotiation.cc:299] Negotiation successful
0925 22:40:34.696980 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 22:40:34.699726 29164 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:40:34.700629 29164 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:40:34.709375 29164 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 190388cc6f5245928266230716e0dd83: Generated new TSK 0
I20260925 22:40:34.710152 29164 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:40:34.718909 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 29075
2026-09-25T22:40:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1370 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790375752797504-7131-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:40:34 dist-test-slave-wjdr krb5kdc[29174](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:40:34 dist-test-slave-wjdr krb5kdc[29174](info): set up 2 sockets
Sep 25 22:40:34 dist-test-slave-wjdr krb5kdc[29174](info): commencing operation
krb5kdc: starting...
W20260925 22:40:35.744830 7131 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Sep 25 22:40:35 dist-test-slave-wjdr krb5kdc[29174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376035, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:40:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:40:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.246.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab.
Entry for principal kudu/127.6.246.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab.
I20260925 22:40:35.865726 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790375752797504-7131-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790375752797504-7131-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790375752797504-7131-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:37207
--webserver_interface=127.6.246.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab
--principal=kudu/127.6.246.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:33125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.246.254:37207
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:40:35.973861 29187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:40:35.974123 29187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:40:35.974165 29187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:40:35.978053 29187 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:40:35.978106 29187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:40:35.978125 29187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:40:35.978142 29187 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:40:35.978156 29187 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:40:35.983301 29187 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:33125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790375752797504-7131-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790375752797504-7131-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.246.254:37207
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab
--principal=kudu/127.6.246.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:37207
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.246.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790375752797504-7131-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:40:35.984340 29187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:40:35.985369 29187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:40:35.994001 29193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:35.994045 29195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:35.994027 29192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:40:35.994160 29187 server_base.cc:1034] running on GCE node
I20260925 22:40:35.994772 29187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:40:35.995776 29187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:40:35.996944 29187 hybrid_clock.cc:648] HybridClock initialized: now 1790376035996918 us; error 38 us; skew 500 ppm
Sep 25 22:40:35 dist-test-slave-wjdr krb5kdc[29174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376035, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:40:35.999661 29187 init.cc:377] Logged in from keytab as kudu/127.6.246.254@KRBTEST.COM (short username kudu)
I20260925 22:40:36.000674 29187 webserver.cc:460] Webserver started at http://127.6.246.254:36499/ using document root <none> and password file <none>
I20260925 22:40:36.001245 29187 fs_manager.cc:359] Metadata directory not provided
I20260925 22:40:36.001317 29187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:40:36.001545 29187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:40:36.003559 29187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790375752797504-7131-0/minicluster-data/master-0/data/instance:
uuid: "4111a986bc3a4f2eaece88ff102d4490"
format_stamp: "Formatted at 2026-09-25 22:40:35 on dist-test-slave-wjdr"
server_key: "575386e9c6544bc897613aa38a3d002c"
server_key_iv: "fa441ddc299407d2e2ffcba440bf82bd"
server_key_version: "encryptionkey@0"
I20260925 22:40:36.004114 29187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790375752797504-7131-0/minicluster-data/master-0/wal/instance:
uuid: "4111a986bc3a4f2eaece88ff102d4490"
format_stamp: "Formatted at 2026-09-25 22:40:35 on dist-test-slave-wjdr"
server_key: "575386e9c6544bc897613aa38a3d002c"
server_key_iv: "fa441ddc299407d2e2ffcba440bf82bd"
server_key_version: "encryptionkey@0"
I20260925 22:40:36.007768 29187 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 22:40:36.010176 29202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:36.011375 29187 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260925 22:40:36.011514 29187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790375752797504-7131-0/minicluster-data/master-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790375752797504-7131-0/minicluster-data/master-0/wal
uuid: "4111a986bc3a4f2eaece88ff102d4490"
format_stamp: "Formatted at 2026-09-25 22:40:35 on dist-test-slave-wjdr"
server_key: "575386e9c6544bc897613aa38a3d002c"
server_key_iv: "fa441ddc299407d2e2ffcba440bf82bd"
server_key_version: "encryptionkey@0"
I20260925 22:40:36.011626 29187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790375752797504-7131-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790375752797504-7131-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790375752797504-7131-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:40:36.032092 29187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:40:36.035305 29187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:40:36.035508 29187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:40:36.043282 29187 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.254:37207
I20260925 22:40:36.043340 29254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.254:37207 every 8 connection(s)
I20260925 22:40:36.044411 29187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
I20260925 22:40:36.048094 29255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:40:36.052007 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 29187
I20260925 22:40:36.052145 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790375752797504-7131-0/minicluster-data/master-0/wal/instance
I20260925 22:40:36.052438 7131 external_mini_cluster.cc:1428] Setting key 7d79acc3ec7e61e2bd4b1089a0172a06
I20260925 22:40:36.055117 29255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4111a986bc3a4f2eaece88ff102d4490: Bootstrap starting.
I20260925 22:40:36.057471 29255 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4111a986bc3a4f2eaece88ff102d4490: Neither blocks nor log segments found. Creating new log.
I20260925 22:40:36.058379 29255 log.cc:824] T 00000000000000000000000000000000 P 4111a986bc3a4f2eaece88ff102d4490: Log is configured to *not* fsync() on all Append() calls
I20260925 22:40:36.060593 29255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4111a986bc3a4f2eaece88ff102d4490: No bootstrap required, opened a new log
I20260925 22:40:36.062170 29258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:36.054181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56881 (local address 127.6.246.254:37207)
0925 22:40:36.054581 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:36.054603 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:36.054648 (+ 45us) server_negotiation.cc:407] Connection header received
0925 22:40:36.055464 (+ 816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:36.055482 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:36.055826 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:36.056103 (+ 277us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:36.056516 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:36.057546 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:36.058795 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:36.059413 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:36.059535 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:36.060374 (+ 839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:36.060379 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:36.060393 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:40:36.060894 (+ 501us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:36.060951 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:40:36.060975 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:36.061057 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:36.061221 (+ 164us) server_negotiation.cc:299] Negotiation successful
0925 22:40:36.061440 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":125,"threads_started":1}
I20260925 22:40:36.064210 29255 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4111a986bc3a4f2eaece88ff102d4490 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4111a986bc3a4f2eaece88ff102d4490" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 37207 } }
I20260925 22:40:36.064447 29255 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4111a986bc3a4f2eaece88ff102d4490 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:40:36.064538 29255 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4111a986bc3a4f2eaece88ff102d4490 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4111a986bc3a4f2eaece88ff102d4490, State: Initialized, Role: FOLLOWER
I20260925 22:40:36.064962 7131 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:40:36.065019 29255 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4111a986bc3a4f2eaece88ff102d4490 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4111a986bc3a4f2eaece88ff102d4490" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 37207 } }
I20260925 22:40:36.065201 29255 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4111a986bc3a4f2eaece88ff102d4490 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:40:36.065266 29255 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4111a986bc3a4f2eaece88ff102d4490 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:40:36.065367 29255 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4111a986bc3a4f2eaece88ff102d4490 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:40:36.066277 29255 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4111a986bc3a4f2eaece88ff102d4490 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4111a986bc3a4f2eaece88ff102d4490" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 37207 } }
I20260925 22:40:36.066625 29255 leader_election.cc:302] T 00000000000000000000000000000000 P 4111a986bc3a4f2eaece88ff102d4490 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4111a986bc3a4f2eaece88ff102d4490; no voters:
I20260925 22:40:36.066978 29255 leader_election.cc:288] T 00000000000000000000000000000000 P 4111a986bc3a4f2eaece88ff102d4490 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:40:36.067075 29264 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4111a986bc3a4f2eaece88ff102d4490 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:40:36.067355 29264 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4111a986bc3a4f2eaece88ff102d4490 [term 1 LEADER]: Becoming Leader. State: Replica: 4111a986bc3a4f2eaece88ff102d4490, State: Running, Role: LEADER
I20260925 22:40:36.067696 29264 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4111a986bc3a4f2eaece88ff102d4490 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4111a986bc3a4f2eaece88ff102d4490" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 37207 } }
I20260925 22:40:36.068216 29255 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4111a986bc3a4f2eaece88ff102d4490 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:40:36.069264 29266 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4111a986bc3a4f2eaece88ff102d4490 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4111a986bc3a4f2eaece88ff102d4490" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4111a986bc3a4f2eaece88ff102d4490" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 37207 } } }
I20260925 22:40:36.069401 29266 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4111a986bc3a4f2eaece88ff102d4490 [sys.catalog]: This master's current role is: LEADER
I20260925 22:40:36.069797 29267 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4111a986bc3a4f2eaece88ff102d4490 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4111a986bc3a4f2eaece88ff102d4490. Latest consensus state: current_term: 1 leader_uuid: "4111a986bc3a4f2eaece88ff102d4490" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4111a986bc3a4f2eaece88ff102d4490" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 37207 } } }
I20260925 22:40:36.069885 29267 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4111a986bc3a4f2eaece88ff102d4490 [sys.catalog]: This master's current role is: LEADER
I20260925 22:40:36.073220 29258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:36.067053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42065 (local address 127.6.246.254:37207)
0925 22:40:36.067165 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:36.067170 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:36.068735 (+ 1565us) server_negotiation.cc:407] Connection header received
0925 22:40:36.068929 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:36.068934 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:36.068990 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:36.069054 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:36.069532 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:36.070313 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:36.071481 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:36.071822 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:36.071945 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:36.072783 (+ 838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:36.072787 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:36.072790 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:40:36.072801 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:36.072826 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:40:36.072845 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:36.072925 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:36.073014 (+ 89us) server_negotiation.cc:299] Negotiation successful
0925 22:40:36.073064 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260925 22:40:36.073405 29280 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4111a986bc3a4f2eaece88ff102d4490: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:40:36.073482 29280 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:40:36.073553 29274 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:40:36.074390 29274 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:40:36.079450 29274 catalog_manager.cc:1347] Generated new cluster ID: 6f955a2f0aff462c81f9545a2c2b788f
I20260925 22:40:36.079535 29274 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:40:36.101503 29274 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:40:36.102761 29274 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:40:36.110536 29274 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4111a986bc3a4f2eaece88ff102d4490: Generated new TSK 0
I20260925 22:40:36.111217 29274 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:40:36.116638 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 29187
2026-09-25T22:40:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1397 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790375752797504-7131-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:40:36 dist-test-slave-wjdr krb5kdc[29286](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:40:36 dist-test-slave-wjdr krb5kdc[29286](info): set up 2 sockets
Sep 25 22:40:36 dist-test-slave-wjdr krb5kdc[29286](info): commencing operation
krb5kdc: starting...
W20260925 22:40:37.160797 7131 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Sep 25 22:40:37 dist-test-slave-wjdr krb5kdc[29286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:40:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:40:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.246.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab.
Entry for principal kudu/127.6.246.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab.
I20260925 22:40:37.281587 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790375752797504-7131-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790375752797504-7131-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790375752797504-7131-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:43819
--webserver_interface=127.6.246.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab
--principal=kudu/127.6.246.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:33345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.246.254:43819
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:40:37.388319 29299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:40:37.388607 29299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:40:37.388650 29299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:40:37.392570 29299 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:40:37.392619 29299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:40:37.392637 29299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:40:37.392654 29299 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:40:37.392668 29299 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:40:37.397835 29299 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:33345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790375752797504-7131-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790375752797504-7131-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.246.254:43819
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab
--principal=kudu/127.6.246.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:43819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.246.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790375752797504-7131-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:40:37.398820 29299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:40:37.399753 29299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:40:37.407770 29304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:37.407770 29305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:37.407797 29307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:40:37.408097 29299 server_base.cc:1034] running on GCE node
I20260925 22:40:37.408689 29299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:40:37.409750 29299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:40:37.410934 29299 hybrid_clock.cc:648] HybridClock initialized: now 1790376037410921 us; error 32 us; skew 500 ppm
Sep 25 22:40:37 dist-test-slave-wjdr krb5kdc[29286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376037, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:40:37.416365 29299 init.cc:377] Logged in from keytab as kudu/127.6.246.254@KRBTEST.COM (short username kudu)
I20260925 22:40:37.417282 29299 webserver.cc:460] Webserver started at http://127.6.246.254:42263/ using document root <none> and password file <none>
I20260925 22:40:37.417821 29299 fs_manager.cc:359] Metadata directory not provided
I20260925 22:40:37.417934 29299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:40:37.418161 29299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:40:37.420114 29299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790375752797504-7131-0/minicluster-data/master-0/data/instance:
uuid: "6d607ec81f414a13b124a37fb2f8a31c"
format_stamp: "Formatted at 2026-09-25 22:40:37 on dist-test-slave-wjdr"
server_key: "6a9612244f1b61e2eab870599a10d7ae"
server_key_iv: "a90ddc3e69ee7671da98e4ca9b106994"
server_key_version: "encryptionkey@0"
I20260925 22:40:37.420730 29299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790375752797504-7131-0/minicluster-data/master-0/wal/instance:
uuid: "6d607ec81f414a13b124a37fb2f8a31c"
format_stamp: "Formatted at 2026-09-25 22:40:37 on dist-test-slave-wjdr"
server_key: "6a9612244f1b61e2eab870599a10d7ae"
server_key_iv: "a90ddc3e69ee7671da98e4ca9b106994"
server_key_version: "encryptionkey@0"
I20260925 22:40:37.424226 29299 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:40:37.426482 29314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:37.427620 29299 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:40:37.427754 29299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790375752797504-7131-0/minicluster-data/master-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790375752797504-7131-0/minicluster-data/master-0/wal
uuid: "6d607ec81f414a13b124a37fb2f8a31c"
format_stamp: "Formatted at 2026-09-25 22:40:37 on dist-test-slave-wjdr"
server_key: "6a9612244f1b61e2eab870599a10d7ae"
server_key_iv: "a90ddc3e69ee7671da98e4ca9b106994"
server_key_version: "encryptionkey@0"
I20260925 22:40:37.427879 29299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790375752797504-7131-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790375752797504-7131-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790375752797504-7131-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:40:37.432319 29299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:40:37.435050 29299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:40:37.435238 29299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:40:37.442032 29299 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.254:43819
I20260925 22:40:37.442088 29366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.254:43819 every 8 connection(s)
I20260925 22:40:37.442986 29299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
I20260925 22:40:37.445719 29367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:40:37.447448 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 29299
I20260925 22:40:37.447598 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790375752797504-7131-0/minicluster-data/master-0/wal/instance
I20260925 22:40:37.447866 7131 external_mini_cluster.cc:1428] Setting key 40bc380e65314bc8c0925a73b03afd84
I20260925 22:40:37.452396 29367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d607ec81f414a13b124a37fb2f8a31c: Bootstrap starting.
I20260925 22:40:37.453261 29370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:37.449251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59219 (local address 127.6.246.254:43819)
0925 22:40:37.449667 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:37.449688 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:37.449728 (+ 40us) server_negotiation.cc:407] Connection header received
0925 22:40:37.450509 (+ 781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:37.450530 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:37.450874 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:37.451239 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:37.451934 (+ 695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:37.451939 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:37.451952 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:40:37.451963 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:37.452023 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:40:37.452049 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:37.452183 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:37.452249 (+ 66us) server_negotiation.cc:299] Negotiation successful
0925 22:40:37.452390 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":113,"threads_started":1}
I20260925 22:40:37.454938 29367 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d607ec81f414a13b124a37fb2f8a31c: Neither blocks nor log segments found. Creating new log.
I20260925 22:40:37.455722 29367 log.cc:824] T 00000000000000000000000000000000 P 6d607ec81f414a13b124a37fb2f8a31c: Log is configured to *not* fsync() on all Append() calls
I20260925 22:40:37.455987 7131 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:40:37.457661 29367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d607ec81f414a13b124a37fb2f8a31c: No bootstrap required, opened a new log
I20260925 22:40:37.459576 29370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:37.458037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40469 (local address 127.6.246.254:43819)
0925 22:40:37.458158 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:37.458162 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:37.458359 (+ 197us) server_negotiation.cc:407] Connection header received
0925 22:40:37.458455 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:37.458456 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:37.458482 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:37.458544 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:37.459214 (+ 670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:37.459217 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:37.459219 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:40:37.459228 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:37.459250 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:40:37.459259 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:37.459323 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:37.459419 (+ 96us) server_negotiation.cc:299] Negotiation successful
0925 22:40:37.459446 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 22:40:37.461923 29367 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d607ec81f414a13b124a37fb2f8a31c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d607ec81f414a13b124a37fb2f8a31c" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 43819 } }
I20260925 22:40:37.462157 29367 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d607ec81f414a13b124a37fb2f8a31c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:40:37.462218 29367 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d607ec81f414a13b124a37fb2f8a31c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d607ec81f414a13b124a37fb2f8a31c, State: Initialized, Role: FOLLOWER
I20260925 22:40:37.462637 29367 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d607ec81f414a13b124a37fb2f8a31c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d607ec81f414a13b124a37fb2f8a31c" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 43819 } }
I20260925 22:40:37.462800 29367 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d607ec81f414a13b124a37fb2f8a31c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:40:37.462863 29367 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d607ec81f414a13b124a37fb2f8a31c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:40:37.462958 29367 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d607ec81f414a13b124a37fb2f8a31c [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:40:37.463904 29367 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d607ec81f414a13b124a37fb2f8a31c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d607ec81f414a13b124a37fb2f8a31c" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 43819 } }
I20260925 22:40:37.464267 29367 leader_election.cc:302] T 00000000000000000000000000000000 P 6d607ec81f414a13b124a37fb2f8a31c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d607ec81f414a13b124a37fb2f8a31c; no voters:
I20260925 22:40:37.464622 29367 leader_election.cc:288] T 00000000000000000000000000000000 P 6d607ec81f414a13b124a37fb2f8a31c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:40:37.464754 29377 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d607ec81f414a13b124a37fb2f8a31c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:40:37.464994 29377 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d607ec81f414a13b124a37fb2f8a31c [term 1 LEADER]: Becoming Leader. State: Replica: 6d607ec81f414a13b124a37fb2f8a31c, State: Running, Role: LEADER
I20260925 22:40:37.465340 29377 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d607ec81f414a13b124a37fb2f8a31c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d607ec81f414a13b124a37fb2f8a31c" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 43819 } }
I20260925 22:40:37.465821 29367 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d607ec81f414a13b124a37fb2f8a31c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:40:37.466967 29379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d607ec81f414a13b124a37fb2f8a31c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d607ec81f414a13b124a37fb2f8a31c. Latest consensus state: current_term: 1 leader_uuid: "6d607ec81f414a13b124a37fb2f8a31c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d607ec81f414a13b124a37fb2f8a31c" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 43819 } } }
I20260925 22:40:37.467103 29379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d607ec81f414a13b124a37fb2f8a31c [sys.catalog]: This master's current role is: LEADER
I20260925 22:40:37.467381 29378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d607ec81f414a13b124a37fb2f8a31c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d607ec81f414a13b124a37fb2f8a31c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d607ec81f414a13b124a37fb2f8a31c" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 43819 } } }
I20260925 22:40:37.467468 29386 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:40:37.467564 29378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d607ec81f414a13b124a37fb2f8a31c [sys.catalog]: This master's current role is: LEADER
I20260925 22:40:37.470820 29386 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:40:37.476342 29386 catalog_manager.cc:1347] Generated new cluster ID: 433ee93f638747b0a6e11cfa0553e336
I20260925 22:40:37.476413 29386 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:40:37.489449 29386 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:40:37.489797 29386 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:40:37.499140 29386 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d607ec81f414a13b124a37fb2f8a31c: Generated new TSK 0
I20260925 22:40:37.499861 29386 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:40:37.504192 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 29299
2026-09-25T22:40:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1387 ms)
[----------] 18 tests from AuthTokenIssuingTest (25756 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790375752797504-7131-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:40:37 dist-test-slave-wjdr krb5kdc[29398](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:40:37 dist-test-slave-wjdr krb5kdc[29398](info): set up 2 sockets
Sep 25 22:40:37 dist-test-slave-wjdr krb5kdc[29398](info): commencing operation
krb5kdc: starting...
W20260925 22:40:38.541430 7131 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Sep 25 22:40:38 dist-test-slave-wjdr krb5kdc[29398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:40:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:40:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.246.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab.
Entry for principal kudu/127.6.246.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab.
I20260925 22:40:38.675326 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790375752797504-7131-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790375752797504-7131-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790375752797504-7131-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:33595
--webserver_interface=127.6.246.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab
--principal=kudu/127.6.246.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:35765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.246.254:33595,127.6.246.253:36333,127.6.246.252:35781
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:40:38.801227 29411 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 22:40:38.801635 29411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:40:38.801710 29411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:40:38.801815 29411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:40:38.806095 29411 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:40:38.806196 29411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:40:38.806237 29411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:40:38.806282 29411 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:40:38.806313 29411 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:40:38.812064 29411 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:35765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790375752797504-7131-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790375752797504-7131-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.246.254:33595,127.6.246.253:36333,127.6.246.252:35781
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab
--principal=kudu/127.6.246.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:33595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.246.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29411
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790375752797504-7131-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:40:38.813544 29411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:40:38.814844 29411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:40:38.826076 29417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:38.826102 29416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:40:38.826229 29411 server_base.cc:1034] running on GCE node
W20260925 22:40:38.826102 29419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:40:38.827275 29411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:40:38.828747 29411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:40:38.829911 29411 hybrid_clock.cc:648] HybridClock initialized: now 1790376038829904 us; error 63 us; skew 500 ppm
Sep 25 22:40:38 dist-test-slave-wjdr krb5kdc[29398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376038, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:40:38.833084 29411 init.cc:377] Logged in from keytab as kudu/127.6.246.254@KRBTEST.COM (short username kudu)
I20260925 22:40:38.834153 29411 webserver.cc:460] Webserver started at http://127.6.246.254:35717/ using document root <none> and password file <none>
I20260925 22:40:38.834820 29411 fs_manager.cc:359] Metadata directory not provided
I20260925 22:40:38.834884 29411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:40:38.835161 29411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:40:38.837608 29411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790375752797504-7131-0/minicluster-data/master-0/data/instance:
uuid: "44f7db4fbd52462ca1f414b9e7f14212"
format_stamp: "Formatted at 2026-09-25 22:40:38 on dist-test-slave-wjdr"
server_key: "27975c54a912b66a61ad134f9c5c559b"
server_key_iv: "72a37d40cb0b14767513f47e00f79008"
server_key_version: "encryptionkey@0"
I20260925 22:40:38.838251 29411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790375752797504-7131-0/minicluster-data/master-0/wal/instance:
uuid: "44f7db4fbd52462ca1f414b9e7f14212"
format_stamp: "Formatted at 2026-09-25 22:40:38 on dist-test-slave-wjdr"
server_key: "27975c54a912b66a61ad134f9c5c559b"
server_key_iv: "72a37d40cb0b14767513f47e00f79008"
server_key_version: "encryptionkey@0"
I20260925 22:40:38.842849 29411 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:40:38.845561 29426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:38.846961 29411 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:40:38.847148 29411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790375752797504-7131-0/minicluster-data/master-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790375752797504-7131-0/minicluster-data/master-0/wal
uuid: "44f7db4fbd52462ca1f414b9e7f14212"
format_stamp: "Formatted at 2026-09-25 22:40:38 on dist-test-slave-wjdr"
server_key: "27975c54a912b66a61ad134f9c5c559b"
server_key_iv: "72a37d40cb0b14767513f47e00f79008"
server_key_version: "encryptionkey@0"
I20260925 22:40:38.847267 29411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790375752797504-7131-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790375752797504-7131-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790375752797504-7131-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:40:38.866349 29411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:40:38.870769 29411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:40:38.871033 29411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:40:38.881460 29411 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.254:33595
I20260925 22:40:38.882311 29478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.254:33595 every 8 connection(s)
I20260925 22:40:38.882884 29411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
I20260925 22:40:38.883533 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 29411
I20260925 22:40:38.883690 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790375752797504-7131-0/minicluster-data/master-0/wal/instance
I20260925 22:40:38.884050 7131 external_mini_cluster.cc:1428] Setting key 0dbd767e83389c404b873965b6767fb1
I20260925 22:40:38.887198 29479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:40:38.892762 29479 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.246.254" port: 33595 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.6.246.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.253@KRBTEST.COM" created.
Sep 25 22:40:38 dist-test-slave-wjdr krb5kdc[29398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376038, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.254@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
I20260925 22:40:38.912662 29483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:38.896202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:33595 (local address 127.0.0.1:37906)
0925 22:40:38.896892 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:38.896935 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:38.898017 (+ 1082us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:38.898996 (+ 979us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:38.899004 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:38.899506 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:40:38.899972 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:38.899988 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:38.902004 (+ 2016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:38.902007 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:38.902860 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:38.902869 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:38.903781 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:38.903784 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:38.903871 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:38.904645 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:40:38.904668 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:40:38.906755 (+ 2087us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:40:38.909396 (+ 2641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:38.909407 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:38.909419 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:38.909741 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:38.910062 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:38.910066 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:38.910068 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:38.910181 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:38.910783 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:40:38.910789 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:40:38.910944 (+ 155us) client_negotiation.cc:769] Sending connection context
0925 22:40:38.911126 (+ 182us) client_negotiation.cc:240] Negotiation successful
0925 22:40:38.911525 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"mutex_wait_us":121,"spinlock_wait_cycles":135168,"thread_start_us":235,"threads_started":1}
I20260925 22:40:38.912662 29482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:38.896202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37906 (local address 127.6.246.254:33595)
0925 22:40:38.897589 (+ 1387us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:38.897601 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:38.897652 (+ 51us) server_negotiation.cc:407] Connection header received
0925 22:40:38.898423 (+ 771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:38.898431 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:38.898718 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:38.898949 (+ 231us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:38.900167 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:38.901877 (+ 1710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:38.902979 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:38.903629 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:38.903817 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:38.906900 (+ 3083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:38.906909 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:38.906924 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:38.906953 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:38.909268 (+ 2315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:38.909848 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:38.909860 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:38.909862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:38.909964 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:38.910275 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:38.910280 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:38.910282 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:38.910673 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:38.910818 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:38.911185 (+ 367us) server_negotiation.cc:299] Negotiation successful
0925 22:40:38.911477 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":1245,"spinlock_wait_cycles":6016,"thread_start_us":210,"threads_started":1}
I20260925 22:40:38.915551 29479 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.246.253" port: 36333 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:40:38.916567 29483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:38.916156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.253:36333 (local address 127.0.0.1:36400)
0925 22:40:38.916363 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:38.916443 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.253:36333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
W20260925 22:40:38.916734 29427 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.246.253:36333: connect: Connection refused (error 111)
I20260925 22:40:38.917757 29483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:38.917478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.253:36333 (local address 127.0.0.1:36410)
0925 22:40:38.917603 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:38.917666 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.253:36333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260925 22:40:38.920413 29479 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.246.253:36333: Network error: Client connection negotiation failed: client connection to 127.6.246.253:36333: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.253.keytab.
Entry for principal kudu/127.6.246.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.253.keytab.
I20260925 22:40:38.925956 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790375752797504-7131-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790375752797504-7131-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790375752797504-7131-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790375752797504-7131-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.253:36333
--webserver_interface=127.6.246.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.253.keytab
--principal=kudu/127.6.246.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:35765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.246.254:33595,127.6.246.253:36333,127.6.246.252:35781
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:40:38.969841 29479 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.246.253" port: 36333 } attempt: 1
I20260925 22:40:38.971086 29483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:38.970738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.253:36333 (local address 127.0.0.1:36422)
0925 22:40:38.970917 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:38.970988 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.253:36333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260925 22:40:38.971901 29483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:38.971673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.253:36333 (local address 127.0.0.1:36438)
0925 22:40:38.971793 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:38.971847 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.253:36333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 22:40:38.972031 29479 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.246.253:36333: Network error: Client connection negotiation failed: client connection to 127.6.246.253:36333: connect: Connection refused (error 111)
I20260925 22:40:39.040205 29479 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.246.253" port: 36333 } attempt: 2
I20260925 22:40:39.041360 29483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:39.040976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.253:36333 (local address 127.0.0.1:36446)
0925 22:40:39.041158 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:39.041237 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.253:36333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260925 22:40:39.042249 29483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:39.042001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.253:36333 (local address 127.0.0.1:36454)
0925 22:40:39.042129 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:39.042192 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.253:36333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260925 22:40:39.042491 29479 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.246.253:36333: Network error: Client connection negotiation failed: client connection to 127.6.246.253:36333: connect: Connection refused (error 111)
W20260925 22:40:39.053014 29486 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 22:40:39.053328 29486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:40:39.053380 29486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:40:39.053483 29486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:40:39.057605 29486 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:40:39.057660 29486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:40:39.057679 29486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:40:39.057696 29486 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:40:39.057732 29486 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:40:39.063192 29486 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:35765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790375752797504-7131-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790375752797504-7131-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.6.246.254:33595,127.6.246.253:36333,127.6.246.252:35781
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.253.keytab
--principal=kudu/127.6.246.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.253:36333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790375752797504-7131-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.246.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29486
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790375752797504-7131-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:40:39.064515 29486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:40:39.065766 29486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:40:39.075244 29492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:39.075256 29491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:39.075270 29494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:40:39.075600 29486 server_base.cc:1034] running on GCE node
I20260925 22:40:39.076251 29486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:40:39.077569 29486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:40:39.078742 29486 hybrid_clock.cc:648] HybridClock initialized: now 1790376039078722 us; error 32 us; skew 500 ppm
Sep 25 22:40:39 dist-test-slave-wjdr krb5kdc[29398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376039, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:40:39.082155 29486 init.cc:377] Logged in from keytab as kudu/127.6.246.253@KRBTEST.COM (short username kudu)
I20260925 22:40:39.083206 29486 webserver.cc:460] Webserver started at http://127.6.246.253:39611/ using document root <none> and password file <none>
I20260925 22:40:39.083837 29486 fs_manager.cc:359] Metadata directory not provided
I20260925 22:40:39.083922 29486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:40:39.084206 29486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:40:39.086565 29486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790375752797504-7131-0/minicluster-data/master-1/data/instance:
uuid: "e25b5cc5f6304ff2aa60566cd7d60250"
format_stamp: "Formatted at 2026-09-25 22:40:39 on dist-test-slave-wjdr"
server_key: "52ab20e47f1f105a6d0697a7681d56ff"
server_key_iv: "9d40938aa5008526800fd93811b11073"
server_key_version: "encryptionkey@0"
I20260925 22:40:39.087178 29486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790375752797504-7131-0/minicluster-data/master-1/wal/instance:
uuid: "e25b5cc5f6304ff2aa60566cd7d60250"
format_stamp: "Formatted at 2026-09-25 22:40:39 on dist-test-slave-wjdr"
server_key: "52ab20e47f1f105a6d0697a7681d56ff"
server_key_iv: "9d40938aa5008526800fd93811b11073"
server_key_version: "encryptionkey@0"
I20260925 22:40:39.091563 29486 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:40:39.094394 29501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:39.095885 29486 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 22:40:39.096047 29486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790375752797504-7131-0/minicluster-data/master-1/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790375752797504-7131-0/minicluster-data/master-1/wal
uuid: "e25b5cc5f6304ff2aa60566cd7d60250"
format_stamp: "Formatted at 2026-09-25 22:40:39 on dist-test-slave-wjdr"
server_key: "52ab20e47f1f105a6d0697a7681d56ff"
server_key_iv: "9d40938aa5008526800fd93811b11073"
server_key_version: "encryptionkey@0"
I20260925 22:40:39.096220 29486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790375752797504-7131-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790375752797504-7131-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790375752797504-7131-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:40:39.122440 29486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:40:39.126276 29486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:40:39.126535 29486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:40:39.133689 29479 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.246.253" port: 36333 } attempt: 3
I20260925 22:40:39.135073 29483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:39.134416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.253:36333 (local address 127.0.0.1:36468)
0925 22:40:39.134811 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:39.134939 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.253:36333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169}
I20260925 22:40:39.135553 29486 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.253:36333
I20260925 22:40:39.135934 29553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.253:36333 every 8 connection(s)
I20260925 22:40:39.137025 29486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790375752797504-7131-0/minicluster-data/master-1/data/info.pb
I20260925 22:40:39.140592 29555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:40:39.143428 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 29486
I20260925 22:40:39.143586 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790375752797504-7131-0/minicluster-data/master-1/wal/instance
I20260925 22:40:39.143916 7131 external_mini_cluster.cc:1428] Setting key 78810ace55353a70472cbd8d42377cd5
Sep 25 22:40:39 dist-test-slave-wjdr krb5kdc[29398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376038, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.254@KRBTEST.COM for kudu/127.6.246.253@KRBTEST.COM
I20260925 22:40:39.145845 29555 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.246.254" port: 33595 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:40:39.149931 29483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:39.135936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.253:36333 (local address 127.0.0.1:36484)
0925 22:40:39.136066 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:39.136085 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:39.136195 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:39.139105 (+ 2910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:39.139108 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:39.139327 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:40:39.139471 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:39.139480 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:39.140926 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:39.140930 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:39.141554 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:39.141564 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:39.142524 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:39.142528 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:39.142641 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:39.143279 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:40:39.143299 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:40:39.145110 (+ 1811us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:40:39.147630 (+ 2520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:39.147637 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:39.147642 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:39.147924 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:39.148387 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:39.148393 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:39.148396 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:39.148537 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:39.149395 (+ 858us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:40:39.149399 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:40:39.149504 (+ 105us) client_negotiation.cc:769] Sending connection context
0925 22:40:39.149619 (+ 115us) client_negotiation.cc:240] Negotiation successful
0925 22:40:39.149766 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260925 22:40:39.151013 29554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:39.137094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36484 (local address 127.6.246.253:36333)
0925 22:40:39.137653 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:39.137673 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:39.137725 (+ 52us) server_negotiation.cc:407] Connection header received
0925 22:40:39.138460 (+ 735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:39.138477 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:39.138809 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:39.139175 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:39.139654 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:39.140797 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:39.141680 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:39.142166 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:39.142585 (+ 419us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:39.145334 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:39.145341 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:39.145356 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:39.145386 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:39.147505 (+ 2119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:39.148066 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:39.148075 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:39.148084 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:39.148253 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:39.148675 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:39.148680 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:39.148685 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:39.149278 (+ 593us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:39.149432 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:39.149776 (+ 344us) server_negotiation.cc:299] Negotiation successful
0925 22:40:39.150083 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":150,"threads_started":1}
I20260925 22:40:39.153041 29479 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.246.252" port: 35781 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.6.246.252@KRBTEST.COM; defaulting to no policy
I20260925 22:40:39.153780 29483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:39.153509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.252:35781 (local address 127.0.0.1:46758)
0925 22:40:39.153657 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:39.153724 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.252:35781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260925 22:40:39.154511 29483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:39.154327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.252:35781 (local address 127.0.0.1:46774)
0925 22:40:39.154421 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:39.154463 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.252:35781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 22:40:39.154706 29479 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.246.252:35781: Network error: Client connection negotiation failed: client connection to 127.6.246.252:35781: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.252@KRBTEST.COM" created.
Sep 25 22:40:39 dist-test-slave-wjdr krb5kdc[29398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376039, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.253@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
I20260925 22:40:39.161857 29558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:39.148035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:33595 (local address 127.0.0.1:37908)
0925 22:40:39.148416 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:39.148489 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:39.148661 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:39.148999 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:39.149004 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:39.149471 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:40:39.149665 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:39.149675 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:39.150913 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:39.150919 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:39.151987 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:39.152001 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:39.153489 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:39.153492 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:39.153606 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:39.154220 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:40:39.154239 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:40:39.157606 (+ 3367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:40:39.160076 (+ 2470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:39.160084 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:39.160088 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:39.160311 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:39.160660 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:39.160662 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:39.160663 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:39.160779 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:39.161313 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:40:39.161318 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:40:39.161421 (+ 103us) client_negotiation.cc:769] Sending connection context
0925 22:40:39.161539 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 22:40:39.161682 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":85,"threads_started":1}
I20260925 22:40:39.162055 29482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:39.148062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37908 (local address 127.6.246.254:33595)
0925 22:40:39.148181 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:39.148185 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:39.148559 (+ 374us) server_negotiation.cc:407] Connection header received
0925 22:40:39.148776 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:39.148779 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:39.148832 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:39.148873 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:39.149842 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:39.150725 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:39.152139 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:39.153241 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:39.153332 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:39.157798 (+ 4466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:39.157802 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:39.157804 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:39.157826 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:39.159954 (+ 2128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:39.160419 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:39.160434 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:39.160439 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:39.160560 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:39.160886 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:39.160893 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:39.160898 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:39.161195 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:39.161326 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:39.161654 (+ 328us) server_negotiation.cc:299] Negotiation successful
0925 22:40:39.161829 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 22:40:39.163163 29555 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.246.253" port: 36333 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 22:40:39 dist-test-slave-wjdr krb5kdc[29398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376039, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.253@KRBTEST.COM for kudu/127.6.246.253@KRBTEST.COM
I20260925 22:40:39.172642 29558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:39.163770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.253:36333 (local address 127.0.0.1:36488)
0925 22:40:39.163880 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:39.163894 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:39.163973 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:39.164329 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:39.164332 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:39.164529 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:40:39.164625 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:39.164631 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:39.165805 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:39.165807 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:39.166343 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:39.166350 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:39.166833 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:39.166835 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:39.166963 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:39.167475 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:40:39.167486 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:40:39.169091 (+ 1605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:40:39.170908 (+ 1817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:39.170911 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:39.170912 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:39.171109 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:39.171470 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:39.171472 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:39.171473 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:39.171556 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:39.172116 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:40:39.172118 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:40:39.172194 (+ 76us) client_negotiation.cc:769] Sending connection context
0925 22:40:39.172285 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 22:40:39.172401 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 22:40:39.172683 29554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:39.163919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36488 (local address 127.6.246.253:36333)
0925 22:40:39.164081 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:39.164088 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:39.164106 (+ 18us) server_negotiation.cc:407] Connection header received
0925 22:40:39.164149 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:39.164156 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:39.164206 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:39.164308 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:39.164743 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:39.165674 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:39.166453 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:39.166700 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:39.166797 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:39.169218 (+ 2421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:39.169221 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:39.169223 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:39.169243 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:39.170794 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:39.171240 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:39.171244 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:39.171245 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:39.171358 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:39.171641 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:39.171644 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:39.171645 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:39.171977 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:39.172101 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:39.172328 (+ 227us) server_negotiation.cc:299] Negotiation successful
0925 22:40:39.172514 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":67}
I20260925 22:40:39.173547 29555 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.246.252" port: 35781 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:40:39.174409 29558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:39.174152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.252:35781 (local address 127.0.0.1:46784)
0925 22:40:39.174277 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:39.174353 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.252:35781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260925 22:40:39.174969 29503 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.246.252:35781: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.252.keytab.
Entry for principal kudu/127.6.246.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.252.keytab.
I20260925 22:40:39.175908 29558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:39.175668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.252:35781 (local address 127.0.0.1:46788)
0925 22:40:39.175808 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:39.175856 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.252:35781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 22:40:39.176136 29555 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.246.252:35781: Network error: Client connection negotiation failed: client connection to 127.6.246.252:35781: connect: Connection refused (error 111)
I20260925 22:40:39.176350 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790375752797504-7131-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790375752797504-7131-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790375752797504-7131-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790375752797504-7131-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.252:35781
--webserver_interface=127.6.246.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.252.keytab
--principal=kudu/127.6.246.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:35765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.246.254:33595,127.6.246.253:36333,127.6.246.252:35781
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:40:39.185869 29479 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.246.252" port: 35781 } attempt: 1
I20260925 22:40:39.186712 29483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:39.186451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.252:35781 (local address 127.0.0.1:46790)
0925 22:40:39.186581 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:39.186642 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.252:35781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260925 22:40:39.187430 29483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:39.187245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.252:35781 (local address 127.0.0.1:46798)
0925 22:40:39.187342 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:39.187384 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.252:35781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260925 22:40:39.187624 29479 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.246.252:35781: Network error: Client connection negotiation failed: client connection to 127.6.246.252:35781: connect: Connection refused (error 111)
I20260925 22:40:39.225425 29555 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.246.252" port: 35781 } attempt: 1
I20260925 22:40:39.226341 29558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:39.226013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.252:35781 (local address 127.0.0.1:46806)
0925 22:40:39.226182 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:39.226253 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.252:35781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260925 22:40:39.227129 29558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:39.226909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.252:35781 (local address 127.0.0.1:46808)
0925 22:40:39.227015 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:39.227067 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.252:35781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260925 22:40:39.227289 29555 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.246.252:35781: Network error: Client connection negotiation failed: client connection to 127.6.246.252:35781: connect: Connection refused (error 111)
I20260925 22:40:39.262811 29479 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.246.252" port: 35781 } attempt: 2
I20260925 22:40:39.263741 29483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:39.263405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.252:35781 (local address 127.0.0.1:46810)
0925 22:40:39.263563 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:39.263624 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.252:35781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260925 22:40:39.264545 29483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:39.264266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.252:35781 (local address 127.0.0.1:46814)
0925 22:40:39.264379 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:39.264435 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.252:35781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 22:40:39.264748 29479 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.246.252:35781: Network error: Client connection negotiation failed: client connection to 127.6.246.252:35781: connect: Connection refused (error 111)
I20260925 22:40:39.295468 29555 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.246.252" port: 35781 } attempt: 2
I20260925 22:40:39.296324 29558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:39.296000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.252:35781 (local address 127.0.0.1:46816)
0925 22:40:39.296174 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:39.296241 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.252:35781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260925 22:40:39.296065 29561 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 22:40:39.296432 29561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:40:39.296519 29561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:40:39.296591 29561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260925 22:40:39.297159 29558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:39.296939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.252:35781 (local address 127.0.0.1:46824)
0925 22:40:39.297046 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:39.297095 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.252:35781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260925 22:40:39.297313 29555 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.246.252:35781: Network error: Client connection negotiation failed: client connection to 127.6.246.252:35781: connect: Connection refused (error 111)
W20260925 22:40:39.301018 29561 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:40:39.301098 29561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:40:39.301143 29561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:40:39.301184 29561 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:40:39.301219 29561 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:40:39.306607 29561 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:35765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790375752797504-7131-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790375752797504-7131-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.6.246.254:33595,127.6.246.253:36333,127.6.246.252:35781
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.252.keytab
--principal=kudu/127.6.246.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.252:35781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790375752797504-7131-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.246.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29561
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790375752797504-7131-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:40:39.307727 29561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:40:39.308835 29561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:40:39.317370 29566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:39.317353 29569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:39.317359 29567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:40:39.317786 29561 server_base.cc:1034] running on GCE node
I20260925 22:40:39.318346 29561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:40:39.319501 29561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:40:39.320662 29561 hybrid_clock.cc:648] HybridClock initialized: now 1790376039320649 us; error 25 us; skew 500 ppm
Sep 25 22:40:39 dist-test-slave-wjdr krb5kdc[29398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376039, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:40:39.323503 29561 init.cc:377] Logged in from keytab as kudu/127.6.246.252@KRBTEST.COM (short username kudu)
I20260925 22:40:39.324501 29561 webserver.cc:460] Webserver started at http://127.6.246.252:37665/ using document root <none> and password file <none>
I20260925 22:40:39.325076 29561 fs_manager.cc:359] Metadata directory not provided
I20260925 22:40:39.325135 29561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:40:39.325366 29561 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:40:39.327441 29561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790375752797504-7131-0/minicluster-data/master-2/data/instance:
uuid: "3995b749ef0a472ea9cda13089d78126"
format_stamp: "Formatted at 2026-09-25 22:40:39 on dist-test-slave-wjdr"
server_key: "bfefa9e33d32e2f818feb97f6be398ed"
server_key_iv: "fdbac133dc8b919fcbd7027d87b27f9e"
server_key_version: "encryptionkey@0"
I20260925 22:40:39.328025 29561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790375752797504-7131-0/minicluster-data/master-2/wal/instance:
uuid: "3995b749ef0a472ea9cda13089d78126"
format_stamp: "Formatted at 2026-09-25 22:40:39 on dist-test-slave-wjdr"
server_key: "bfefa9e33d32e2f818feb97f6be398ed"
server_key_iv: "fdbac133dc8b919fcbd7027d87b27f9e"
server_key_version: "encryptionkey@0"
I20260925 22:40:39.331962 29561 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 22:40:39.334616 29576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:39.335908 29561 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:40:39.336035 29561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790375752797504-7131-0/minicluster-data/master-2/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790375752797504-7131-0/minicluster-data/master-2/wal
uuid: "3995b749ef0a472ea9cda13089d78126"
format_stamp: "Formatted at 2026-09-25 22:40:39 on dist-test-slave-wjdr"
server_key: "bfefa9e33d32e2f818feb97f6be398ed"
server_key_iv: "fdbac133dc8b919fcbd7027d87b27f9e"
server_key_version: "encryptionkey@0"
I20260925 22:40:39.336148 29561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790375752797504-7131-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790375752797504-7131-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790375752797504-7131-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:40:39.363940 29479 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.246.252" port: 35781 } attempt: 3
I20260925 22:40:39.364866 29483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:39.364545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.252:35781 (local address 127.0.0.1:46832)
0925 22:40:39.364696 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:39.364782 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.252:35781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260925 22:40:39.365625 29483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:39.365410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.252:35781 (local address 127.0.0.1:46836)
0925 22:40:39.365524 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:39.365575 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.252:35781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 22:40:39.365769 29479 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.246.252:35781: Network error: Client connection negotiation failed: client connection to 127.6.246.252:35781: connect: Connection refused (error 111)
I20260925 22:40:39.376727 29561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:40:39.380095 29561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:40:39.380304 29561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:40:39.387394 29561 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.252:35781
I20260925 22:40:39.387684 29628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.252:35781 every 8 connection(s)
I20260925 22:40:39.388360 29561 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790375752797504-7131-0/minicluster-data/master-2/data/info.pb
I20260925 22:40:39.388502 29555 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.246.252" port: 35781 } attempt: 3
I20260925 22:40:39.391566 29629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:40:39.393189 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 29561
I20260925 22:40:39.393321 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790375752797504-7131-0/minicluster-data/master-2/wal/instance
I20260925 22:40:39.393630 7131 external_mini_cluster.cc:1428] Setting key 95c583c91718c8d232d4935541c9b2c7
I20260925 22:40:39.396636 29629 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.246.254" port: 33595 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 22:40:39 dist-test-slave-wjdr krb5kdc[29398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376039, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.253@KRBTEST.COM for kudu/127.6.246.252@KRBTEST.COM
Sep 25 22:40:39 dist-test-slave-wjdr krb5kdc[29398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.246.252@KRBTEST.COM
I20260925 22:40:39.402707 29558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:39.389062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.252:35781 (local address 127.0.0.1:46846)
0925 22:40:39.389216 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:39.389233 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:39.389345 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:39.391625 (+ 2280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:39.391628 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:39.391833 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:40:39.391939 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:39.391946 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:39.393452 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:39.393456 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:39.394193 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:39.394205 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:39.395155 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:39.395159 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:39.395270 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:39.395840 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:40:39.395852 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:40:39.397313 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:40:39.399884 (+ 2571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:39.399887 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:39.399890 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:39.400184 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:39.400721 (+ 537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:39.400725 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:39.400727 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:39.400829 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:39.402110 (+ 1281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:40:39.402117 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:40:39.402232 (+ 115us) client_negotiation.cc:769] Sending connection context
0925 22:40:39.402363 (+ 131us) client_negotiation.cc:240] Negotiation successful
0925 22:40:39.402522 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260925 22:40:39.403584 29631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:39.389891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46846 (local address 127.6.246.252:35781)
0925 22:40:39.390326 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:39.390341 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:39.390376 (+ 35us) server_negotiation.cc:407] Connection header received
0925 22:40:39.391017 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:39.391033 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:39.391353 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:39.391679 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:39.392111 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:39.393273 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:39.394344 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:39.395011 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:39.395176 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:39.397499 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:39.397504 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:39.397517 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:39.397542 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:39.399759 (+ 2217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:39.400361 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:39.400369 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:39.400377 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:39.400573 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:39.400940 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:39.400943 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:39.400945 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:39.401992 (+ 1047us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:39.402122 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:39.402517 (+ 395us) server_negotiation.cc:299] Negotiation successful
0925 22:40:39.402830 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":509,"server-negotiator.queue_time_us":287,"thread_start_us":139,"threads_started":1}
Sep 25 22:40:39 dist-test-slave-wjdr krb5kdc[29398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376039, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.252@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
I20260925 22:40:39.407306 29632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:39.394314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46852 (local address 127.6.246.252:35781)
0925 22:40:39.394646 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:39.394653 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:39.394830 (+ 177us) server_negotiation.cc:407] Connection header received
0925 22:40:39.394958 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:39.394962 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:39.395011 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:39.395104 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:39.396390 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:39.397248 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:39.398393 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:39.398692 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:39.398810 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:39.401993 (+ 3183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:39.401997 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:39.401999 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:39.402025 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:39.405120 (+ 3095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:39.405806 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:39.405810 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:39.405811 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:39.405898 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:39.406207 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:39.406212 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:39.406214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:39.406392 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:39.406578 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:39.406970 (+ 392us) server_negotiation.cc:299] Negotiation successful
0925 22:40:39.407078 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":125,"threads_started":1}
I20260925 22:40:39.408434 29555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e25b5cc5f6304ff2aa60566cd7d60250: Bootstrap starting.
I20260925 22:40:39.409267 29634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:39.398803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:33595 (local address 127.0.0.1:37910)
0925 22:40:39.399193 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:39.399261 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:39.399377 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:39.399687 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:39.399693 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:39.400151 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:40:39.400374 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:39.400381 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:39.401333 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:39.401336 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:39.402023 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:39.402034 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:39.402689 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:39.402692 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:39.402817 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:39.403330 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:40:39.403341 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:40:39.405421 (+ 2080us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:40:39.407463 (+ 2042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:39.407471 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:39.407476 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:39.407763 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:39.408117 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:39.408120 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:39.408121 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:39.408217 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:39.408687 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:40:39.408691 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:40:39.408791 (+ 100us) client_negotiation.cc:769] Sending connection context
0925 22:40:39.408932 (+ 141us) client_negotiation.cc:240] Negotiation successful
0925 22:40:39.409101 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":112,"threads_started":1}
I20260925 22:40:39.409260 29482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:39.398621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37910 (local address 127.6.246.254:33595)
0925 22:40:39.398772 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:39.398779 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:39.399325 (+ 546us) server_negotiation.cc:407] Connection header received
0925 22:40:39.399499 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:39.399502 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:39.399579 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:39.399657 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:39.400511 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:39.401213 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:39.402198 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:39.402549 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:39.402652 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:39.405611 (+ 2959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:39.405615 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:39.405618 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:39.405649 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:39.407333 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:39.407867 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:39.407870 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:39.407871 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:39.407945 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:39.408312 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:39.408314 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:39.408315 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:39.408534 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:39.408639 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:39.408990 (+ 351us) server_negotiation.cc:299] Negotiation successful
0925 22:40:39.409100 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 22:40:39.410494 29629 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.246.253" port: 36333 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:40:39.411649 29555 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e25b5cc5f6304ff2aa60566cd7d60250: Neither blocks nor log segments found. Creating new log.
I20260925 22:40:39.412536 29555 log.cc:824] T 00000000000000000000000000000000 P e25b5cc5f6304ff2aa60566cd7d60250: Log is configured to *not* fsync() on all Append() calls
I20260925 22:40:39.414809 29555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e25b5cc5f6304ff2aa60566cd7d60250: No bootstrap required, opened a new log
Sep 25 22:40:39 dist-test-slave-wjdr krb5kdc[29398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376039, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.252@KRBTEST.COM for kudu/127.6.246.253@KRBTEST.COM
Sep 25 22:40:39 dist-test-slave-wjdr krb5kdc[29398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
I20260925 22:40:39.419457 29634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:39.410954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.253:36333 (local address 127.0.0.1:36498)
0925 22:40:39.411098 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:39.411110 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:39.411181 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:39.411445 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:39.411447 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:39.411602 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:40:39.411684 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:39.411688 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:39.412759 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:39.412762 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:39.413529 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:39.413539 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:39.413997 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:39.414001 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:39.414091 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:39.414567 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:40:39.414584 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:40:39.415895 (+ 1311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:40:39.417901 (+ 2006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:39.417904 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:39.417905 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:39.418087 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:39.418403 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:39.418407 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:39.418410 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:39.418493 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:39.418956 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:40:39.418959 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:40:39.419057 (+ 98us) client_negotiation.cc:769] Sending connection context
0925 22:40:39.419172 (+ 115us) client_negotiation.cc:240] Negotiation successful
0925 22:40:39.419304 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20260925 22:40:39.419688 29554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:39.411020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36498 (local address 127.6.246.253:36333)
0925 22:40:39.411129 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:39.411136 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:39.411151 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:40:39.411298 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:39.411301 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:39.411349 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:39.411431 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:39.411869 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:39.412636 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:39.413652 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:39.413886 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:39.413994 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:39.416024 (+ 2030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:39.416028 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:39.416030 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:39.416053 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:39.417610 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:39.418218 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:39.418221 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:39.418223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:39.418310 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:39.418582 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:39.418587 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:39.418588 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:39.418838 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:39.418945 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:39.419250 (+ 305us) server_negotiation.cc:299] Negotiation successful
0925 22:40:39.419369 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 22:40:39.419559 29555 raft_consensus.cc:348] T 00000000000000000000000000000000 P e25b5cc5f6304ff2aa60566cd7d60250 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f7db4fbd52462ca1f414b9e7f14212" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 33595 } } peers { permanent_uuid: "e25b5cc5f6304ff2aa60566cd7d60250" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 36333 } } peers { permanent_uuid: "3995b749ef0a472ea9cda13089d78126" member_type: VOTER last_known_addr { host: "127.6.246.252" port: 35781 } }
I20260925 22:40:39.419759 29555 raft_consensus.cc:729] T 00000000000000000000000000000000 P e25b5cc5f6304ff2aa60566cd7d60250 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e25b5cc5f6304ff2aa60566cd7d60250, State: Initialized, Role: FOLLOWER
I20260925 22:40:39.420113 29555 consensus_queue.cc:260] T 00000000000000000000000000000000 P e25b5cc5f6304ff2aa60566cd7d60250 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f7db4fbd52462ca1f414b9e7f14212" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 33595 } } peers { permanent_uuid: "e25b5cc5f6304ff2aa60566cd7d60250" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 36333 } } peers { permanent_uuid: "3995b749ef0a472ea9cda13089d78126" member_type: VOTER last_known_addr { host: "127.6.246.252" port: 35781 } }
I20260925 22:40:39.420367 29629 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.246.252" port: 35781 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:40:39.420867 29482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:39.410328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37922 (local address 127.6.246.254:33595)
0925 22:40:39.410433 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:39.410436 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:39.410929 (+ 493us) server_negotiation.cc:407] Connection header received
0925 22:40:39.411117 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:39.411120 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:39.411163 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:39.411211 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:39.412083 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:39.412806 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:39.413779 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:39.414090 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:39.414166 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:39.416928 (+ 2762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:39.416932 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:39.416935 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:39.416959 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:39.418587 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:39.419028 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:39.419032 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:39.419034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:39.419116 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:39.419466 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:39.419470 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:39.419472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:39.419703 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:39.419901 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:39.420551 (+ 650us) server_negotiation.cc:299] Negotiation successful
0925 22:40:39.420682 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 22:40:39.421603 29555 sys_catalog.cc:564] T 00000000000000000000000000000000 P e25b5cc5f6304ff2aa60566cd7d60250 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:40:39.424044 29636 sys_catalog.cc:455] T 00000000000000000000000000000000 P e25b5cc5f6304ff2aa60566cd7d60250 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f7db4fbd52462ca1f414b9e7f14212" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 33595 } } peers { permanent_uuid: "e25b5cc5f6304ff2aa60566cd7d60250" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 36333 } } peers { permanent_uuid: "3995b749ef0a472ea9cda13089d78126" member_type: VOTER last_known_addr { host: "127.6.246.252" port: 35781 } } }
I20260925 22:40:39.424166 29636 sys_catalog.cc:458] T 00000000000000000000000000000000 P e25b5cc5f6304ff2aa60566cd7d60250 [sys.catalog]: This master's current role is: FOLLOWER
Sep 25 22:40:39 dist-test-slave-wjdr krb5kdc[29398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376039, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.252@KRBTEST.COM for kudu/127.6.246.252@KRBTEST.COM
W20260925 22:40:39.427138 29647 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e25b5cc5f6304ff2aa60566cd7d60250: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:40:39.427213 29647 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 25 22:40:39 dist-test-slave-wjdr krb5kdc[29398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.246.253@KRBTEST.COM
I20260925 22:40:39.429879 29634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:39.420870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.252:35781 (local address 127.0.0.1:46856)
0925 22:40:39.420997 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:39.421013 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:39.421087 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:39.421446 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:39.421449 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:39.421607 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:40:39.421687 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:39.421693 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:39.422722 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:39.422727 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:39.423557 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:39.423568 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:39.424391 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:39.424394 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:39.424491 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:39.424979 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:40:39.424993 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:40:39.426408 (+ 1415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:40:39.428171 (+ 1763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:39.428174 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:39.428175 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:39.428415 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:39.428744 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:39.428749 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:39.428753 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:39.428847 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:39.429350 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:40:39.429353 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:40:39.429450 (+ 97us) client_negotiation.cc:769] Sending connection context
0925 22:40:39.429562 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 22:40:39.429701 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260925 22:40:39.429889 29632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:39.421155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46856 (local address 127.6.246.252:35781)
0925 22:40:39.421254 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:39.421258 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:39.421271 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:40:39.421304 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:39.421307 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:39.421350 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:39.421437 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:39.421813 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:39.422594 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:39.423704 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:39.424053 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:39.424150 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:39.426555 (+ 2405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:39.426559 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:39.426561 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:39.426590 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:39.428051 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:39.428543 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:39.428547 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:39.428549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:39.428635 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:39.428953 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:39.428959 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:39.428962 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:39.429232 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:39.429338 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:39.429604 (+ 266us) server_negotiation.cc:299] Negotiation successful
0925 22:40:39.429765 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":39}
I20260925 22:40:39.425804 29486 master_runner.cc:186] Error getting master registration for 127.6.246.254:33595: OK, instance_id { permanent_uuid: "44f7db4fbd52462ca1f414b9e7f14212" instance_seqno: 1790376038879067 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 22:40:39.430495 29486 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 22:40:39.432674 29629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3995b749ef0a472ea9cda13089d78126: Bootstrap starting.
I20260925 22:40:39.433413 29554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:39.422551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36506 (local address 127.6.246.253:36333)
0925 22:40:39.422856 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:39.422860 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:39.422878 (+ 18us) server_negotiation.cc:407] Connection header received
0925 22:40:39.422905 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:39.422907 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:39.422938 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:39.423024 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:39.423816 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:39.424555 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:39.425555 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:39.425872 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:39.425943 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:39.429557 (+ 3614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:39.429561 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:39.429564 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:39.429589 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:39.431417 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:39.431923 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:39.431929 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:39.431932 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:39.432017 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:39.432334 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:39.432338 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:39.432340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:39.432597 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:39.432766 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:39.433148 (+ 382us) server_negotiation.cc:299] Negotiation successful
0925 22:40:39.433267 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238}
I20260925 22:40:39.435163 29629 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3995b749ef0a472ea9cda13089d78126: Neither blocks nor log segments found. Creating new log.
I20260925 22:40:39.436002 29629 log.cc:824] T 00000000000000000000000000000000 P 3995b749ef0a472ea9cda13089d78126: Log is configured to *not* fsync() on all Append() calls
I20260925 22:40:39.437348 7131 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:40:39.438030 29629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3995b749ef0a472ea9cda13089d78126: No bootstrap required, opened a new log
I20260925 22:40:39.438912 29453 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "44f7db4fbd52462ca1f414b9e7f14212"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37922
I20260925 22:40:39.442301 29629 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3995b749ef0a472ea9cda13089d78126 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f7db4fbd52462ca1f414b9e7f14212" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 33595 } } peers { permanent_uuid: "e25b5cc5f6304ff2aa60566cd7d60250" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 36333 } } peers { permanent_uuid: "3995b749ef0a472ea9cda13089d78126" member_type: VOTER last_known_addr { host: "127.6.246.252" port: 35781 } }
I20260925 22:40:39.442587 29629 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3995b749ef0a472ea9cda13089d78126 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3995b749ef0a472ea9cda13089d78126, State: Initialized, Role: FOLLOWER
I20260925 22:40:39.442900 29629 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3995b749ef0a472ea9cda13089d78126 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f7db4fbd52462ca1f414b9e7f14212" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 33595 } } peers { permanent_uuid: "e25b5cc5f6304ff2aa60566cd7d60250" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 36333 } } peers { permanent_uuid: "3995b749ef0a472ea9cda13089d78126" member_type: VOTER last_known_addr { host: "127.6.246.252" port: 35781 } }
I20260925 22:40:39.443369 29650 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3995b749ef0a472ea9cda13089d78126 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f7db4fbd52462ca1f414b9e7f14212" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 33595 } } peers { permanent_uuid: "e25b5cc5f6304ff2aa60566cd7d60250" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 36333 } } peers { permanent_uuid: "3995b749ef0a472ea9cda13089d78126" member_type: VOTER last_known_addr { host: "127.6.246.252" port: 35781 } } }
I20260925 22:40:39.443497 29650 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3995b749ef0a472ea9cda13089d78126 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:40:39.444126 29629 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3995b749ef0a472ea9cda13089d78126 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 22:40:39.449283 29661 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3995b749ef0a472ea9cda13089d78126: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:40:39.449358 29661 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:40:39.447691 29561 master_runner.cc:186] Error getting master registration for 127.6.246.254:33595: OK, instance_id { permanent_uuid: "44f7db4fbd52462ca1f414b9e7f14212" instance_seqno: 1790376038879067 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 22:40:39.450855 29561 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 22:40:39.529945 29479 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.246.252" port: 35781 } attempt: 4
Sep 25 22:40:39 dist-test-slave-wjdr krb5kdc[29398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376038, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.254@KRBTEST.COM for kudu/127.6.246.252@KRBTEST.COM
I20260925 22:40:39.551852 29483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:39.530619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.252:35781 (local address 127.0.0.1:46870)
0925 22:40:39.530833 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:39.530847 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:39.530963 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:39.531225 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:39.531228 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:39.531451 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:40:39.531558 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:39.531565 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:39.541635 (+ 10070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:39.541640 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:39.542534 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:39.542548 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:39.543267 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:39.543271 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:39.543354 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:39.544568 (+ 1214us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:40:39.544583 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:40:39.546218 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:40:39.549997 (+ 3779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:39.550001 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:39.550003 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:39.550272 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:39.550635 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:39.550638 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:39.550640 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:39.550728 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:39.551275 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:40:39.551279 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:40:39.551375 (+ 96us) client_negotiation.cc:769] Sending connection context
0925 22:40:39.551472 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 22:40:39.551659 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133}
I20260925 22:40:39.552389 29632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:39.530667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46870 (local address 127.6.246.252:35781)
0925 22:40:39.530835 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:39.530843 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:39.530900 (+ 57us) server_negotiation.cc:407] Connection header received
0925 22:40:39.531065 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:39.531068 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:39.531124 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:39.531204 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:39.540612 (+ 9408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:39.541452 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:39.542684 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:39.543028 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:39.543121 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:39.546414 (+ 3293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:39.546418 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:39.546421 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:39.546450 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:39.549858 (+ 3408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:39.550386 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:39.550390 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:39.550392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:39.550533 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:39.550843 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:39.550896 (+ 53us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:39.550898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:39.551137 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:39.551229 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:39.552120 (+ 891us) server_negotiation.cc:299] Negotiation successful
0925 22:40:39.552244 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 22:40:39.555790 29479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44f7db4fbd52462ca1f414b9e7f14212: Bootstrap starting.
I20260925 22:40:39.558604 29479 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44f7db4fbd52462ca1f414b9e7f14212: Neither blocks nor log segments found. Creating new log.
I20260925 22:40:39.559453 29479 log.cc:824] T 00000000000000000000000000000000 P 44f7db4fbd52462ca1f414b9e7f14212: Log is configured to *not* fsync() on all Append() calls
I20260925 22:40:39.561655 29479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44f7db4fbd52462ca1f414b9e7f14212: No bootstrap required, opened a new log
I20260925 22:40:39.564826 29479 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44f7db4fbd52462ca1f414b9e7f14212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f7db4fbd52462ca1f414b9e7f14212" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 33595 } } peers { permanent_uuid: "e25b5cc5f6304ff2aa60566cd7d60250" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 36333 } } peers { permanent_uuid: "3995b749ef0a472ea9cda13089d78126" member_type: VOTER last_known_addr { host: "127.6.246.252" port: 35781 } }
I20260925 22:40:39.565141 29479 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44f7db4fbd52462ca1f414b9e7f14212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44f7db4fbd52462ca1f414b9e7f14212, State: Initialized, Role: FOLLOWER
I20260925 22:40:39.565516 29479 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44f7db4fbd52462ca1f414b9e7f14212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f7db4fbd52462ca1f414b9e7f14212" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 33595 } } peers { permanent_uuid: "e25b5cc5f6304ff2aa60566cd7d60250" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 36333 } } peers { permanent_uuid: "3995b749ef0a472ea9cda13089d78126" member_type: VOTER last_known_addr { host: "127.6.246.252" port: 35781 } }
I20260925 22:40:39.566183 29664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44f7db4fbd52462ca1f414b9e7f14212 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f7db4fbd52462ca1f414b9e7f14212" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 33595 } } peers { permanent_uuid: "e25b5cc5f6304ff2aa60566cd7d60250" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 36333 } } peers { permanent_uuid: "3995b749ef0a472ea9cda13089d78126" member_type: VOTER last_known_addr { host: "127.6.246.252" port: 35781 } } }
I20260925 22:40:39.566458 29664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44f7db4fbd52462ca1f414b9e7f14212 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:40:39.567024 29479 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44f7db4fbd52462ca1f414b9e7f14212 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:40:39.567093 29453 raft_consensus.cc:482] T 00000000000000000000000000000000 P 44f7db4fbd52462ca1f414b9e7f14212 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 22:40:39.567212 29453 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44f7db4fbd52462ca1f414b9e7f14212 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:40:39.568573 29453 raft_consensus.cc:504] T 00000000000000000000000000000000 P 44f7db4fbd52462ca1f414b9e7f14212 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f7db4fbd52462ca1f414b9e7f14212" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 33595 } } peers { permanent_uuid: "e25b5cc5f6304ff2aa60566cd7d60250" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 36333 } } peers { permanent_uuid: "3995b749ef0a472ea9cda13089d78126" member_type: VOTER last_known_addr { host: "127.6.246.252" port: 35781 } }
I20260925 22:40:39.569543 29453 leader_election.cc:288] T 00000000000000000000000000000000 P 44f7db4fbd52462ca1f414b9e7f14212 [CANDIDATE]: Term 1 election: Requested vote from peers e25b5cc5f6304ff2aa60566cd7d60250 (127.6.246.253:36333), 3995b749ef0a472ea9cda13089d78126 (127.6.246.252:35781)
I20260925 22:40:39.571116 29528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "44f7db4fbd52462ca1f414b9e7f14212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e25b5cc5f6304ff2aa60566cd7d60250"
I20260925 22:40:39.571362 29528 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e25b5cc5f6304ff2aa60566cd7d60250 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:40:39.572899 29528 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e25b5cc5f6304ff2aa60566cd7d60250 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44f7db4fbd52462ca1f414b9e7f14212 in term 1.
W20260925 22:40:39.573173 29675 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 44f7db4fbd52462ca1f414b9e7f14212: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:40:39.573238 29675 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:40:39.573364 29603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "44f7db4fbd52462ca1f414b9e7f14212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3995b749ef0a472ea9cda13089d78126"
I20260925 22:40:39.573592 29603 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3995b749ef0a472ea9cda13089d78126 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:40:39.574958 29603 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3995b749ef0a472ea9cda13089d78126 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44f7db4fbd52462ca1f414b9e7f14212 in term 1.
I20260925 22:40:39.575459 29428 leader_election.cc:302] T 00000000000000000000000000000000 P 44f7db4fbd52462ca1f414b9e7f14212 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3995b749ef0a472ea9cda13089d78126, 44f7db4fbd52462ca1f414b9e7f14212; no voters:
I20260925 22:40:39.579144 29664 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 44f7db4fbd52462ca1f414b9e7f14212 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:40:39.579459 29664 raft_consensus.cc:686] T 00000000000000000000000000000000 P 44f7db4fbd52462ca1f414b9e7f14212 [term 1 LEADER]: Becoming Leader. State: Replica: 44f7db4fbd52462ca1f414b9e7f14212, State: Running, Role: LEADER
I20260925 22:40:39.579879 29664 consensus_queue.cc:237] T 00000000000000000000000000000000 P 44f7db4fbd52462ca1f414b9e7f14212 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f7db4fbd52462ca1f414b9e7f14212" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 33595 } } peers { permanent_uuid: "e25b5cc5f6304ff2aa60566cd7d60250" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 36333 } } peers { permanent_uuid: "3995b749ef0a472ea9cda13089d78126" member_type: VOTER last_known_addr { host: "127.6.246.252" port: 35781 } }
I20260925 22:40:39.582026 29664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44f7db4fbd52462ca1f414b9e7f14212 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44f7db4fbd52462ca1f414b9e7f14212. Latest consensus state: current_term: 1 leader_uuid: "44f7db4fbd52462ca1f414b9e7f14212" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f7db4fbd52462ca1f414b9e7f14212" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 33595 } } peers { permanent_uuid: "e25b5cc5f6304ff2aa60566cd7d60250" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 36333 } } peers { permanent_uuid: "3995b749ef0a472ea9cda13089d78126" member_type: VOTER last_known_addr { host: "127.6.246.252" port: 35781 } } }
I20260925 22:40:39.582201 29664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44f7db4fbd52462ca1f414b9e7f14212 [sys.catalog]: This master's current role is: LEADER
I20260925 22:40:39.582679 29411 master_runner.cc:263] No leader master found from master 44f7db4fbd52462ca1f414b9e7f14212
I20260925 22:40:39.582758 29411 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 22:40:39.583539 29685 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:40:39.584609 29685 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:40:39.597833 29554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:39.577905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36512 (local address 127.6.246.253:36333)
0925 22:40:39.578025 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:39.578030 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:39.583410 (+ 5380us) server_negotiation.cc:407] Connection header received
0925 22:40:39.583599 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:39.583602 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:39.583649 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:39.583698 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:39.584809 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:39.585533 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:39.586695 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:39.587034 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:39.587111 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:39.592730 (+ 5619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:39.592734 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:39.592737 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:39.592776 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:39.594611 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:39.596389 (+ 1778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:39.596393 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:39.596395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:39.596517 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:39.596837 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:39.596840 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:39.596842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:39.597058 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:39.597169 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:39.597511 (+ 342us) server_negotiation.cc:299] Negotiation successful
0925 22:40:39.597638 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 22:40:39.602574 29603 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3995b749ef0a472ea9cda13089d78126 [term 1 FOLLOWER]: Refusing update from remote peer 44f7db4fbd52462ca1f414b9e7f14212: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:40:39.603560 29528 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e25b5cc5f6304ff2aa60566cd7d60250 [term 1 FOLLOWER]: Refusing update from remote peer 44f7db4fbd52462ca1f414b9e7f14212: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:40:39.604233 29683 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 44f7db4fbd52462ca1f414b9e7f14212 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e25b5cc5f6304ff2aa60566cd7d60250" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 36333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:40:39.604768 29664 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 44f7db4fbd52462ca1f414b9e7f14212 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3995b749ef0a472ea9cda13089d78126" member_type: VOTER last_known_addr { host: "127.6.246.252" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:40:39.609620 29632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:39.582759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46884 (local address 127.6.246.252:35781)
0925 22:40:39.582919 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:39.582925 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:39.590224 (+ 7299us) server_negotiation.cc:407] Connection header received
0925 22:40:39.590398 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:39.590401 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:39.590448 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:39.590493 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:39.591287 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:39.591996 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:39.593083 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:39.593467 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:39.593550 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:39.605747 (+ 12197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:39.605753 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:39.605756 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:39.605787 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:39.607485 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:39.607998 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:39.608001 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:39.608003 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:39.608084 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:39.608407 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:39.608413 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:39.608417 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:39.608682 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:39.608798 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:39.609314 (+ 516us) server_negotiation.cc:299] Negotiation successful
0925 22:40:39.609442 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260925 22:40:39.610006 29636 sys_catalog.cc:455] T 00000000000000000000000000000000 P e25b5cc5f6304ff2aa60566cd7d60250 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44f7db4fbd52462ca1f414b9e7f14212. Latest consensus state: current_term: 1 leader_uuid: "44f7db4fbd52462ca1f414b9e7f14212" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f7db4fbd52462ca1f414b9e7f14212" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 33595 } } peers { permanent_uuid: "e25b5cc5f6304ff2aa60566cd7d60250" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 36333 } } peers { permanent_uuid: "3995b749ef0a472ea9cda13089d78126" member_type: VOTER last_known_addr { host: "127.6.246.252" port: 35781 } } }
I20260925 22:40:39.610148 29636 sys_catalog.cc:458] T 00000000000000000000000000000000 P e25b5cc5f6304ff2aa60566cd7d60250 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:40:39.613664 29650 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3995b749ef0a472ea9cda13089d78126 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44f7db4fbd52462ca1f414b9e7f14212. Latest consensus state: current_term: 1 leader_uuid: "44f7db4fbd52462ca1f414b9e7f14212" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f7db4fbd52462ca1f414b9e7f14212" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 33595 } } peers { permanent_uuid: "e25b5cc5f6304ff2aa60566cd7d60250" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 36333 } } peers { permanent_uuid: "3995b749ef0a472ea9cda13089d78126" member_type: VOTER last_known_addr { host: "127.6.246.252" port: 35781 } } }
I20260925 22:40:39.613812 29650 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3995b749ef0a472ea9cda13089d78126 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:40:39.614669 29482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:39.577039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37936 (local address 127.6.246.254:33595)
0925 22:40:39.577259 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:39.577263 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:39.588789 (+ 11526us) server_negotiation.cc:407] Connection header received
0925 22:40:39.588909 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:39.588912 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:39.588959 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:39.589038 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:39.589876 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:39.590557 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:39.591663 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:39.591992 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:39.592079 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:39.599620 (+ 7541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:39.599624 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:39.599627 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:39.599657 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:39.601631 (+ 1974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:39.613145 (+ 11514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:39.613150 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:39.613151 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:39.613259 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:39.613629 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:39.613634 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:39.613637 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:39.613917 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:39.614039 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:39.614390 (+ 351us) server_negotiation.cc:299] Negotiation successful
0925 22:40:39.614520 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144}
I20260925 22:40:39.617233 29683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44f7db4fbd52462ca1f414b9e7f14212 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "44f7db4fbd52462ca1f414b9e7f14212" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f7db4fbd52462ca1f414b9e7f14212" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 33595 } } peers { permanent_uuid: "e25b5cc5f6304ff2aa60566cd7d60250" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 36333 } } peers { permanent_uuid: "3995b749ef0a472ea9cda13089d78126" member_type: VOTER last_known_addr { host: "127.6.246.252" port: 35781 } } }
I20260925 22:40:39.617375 29683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44f7db4fbd52462ca1f414b9e7f14212 [sys.catalog]: This master's current role is: LEADER
I20260925 22:40:39.618562 29636 sys_catalog.cc:455] T 00000000000000000000000000000000 P e25b5cc5f6304ff2aa60566cd7d60250 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "44f7db4fbd52462ca1f414b9e7f14212" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f7db4fbd52462ca1f414b9e7f14212" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 33595 } } peers { permanent_uuid: "e25b5cc5f6304ff2aa60566cd7d60250" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 36333 } } peers { permanent_uuid: "3995b749ef0a472ea9cda13089d78126" member_type: VOTER last_known_addr { host: "127.6.246.252" port: 35781 } } }
I20260925 22:40:39.618698 29636 sys_catalog.cc:458] T 00000000000000000000000000000000 P e25b5cc5f6304ff2aa60566cd7d60250 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:40:39.620599 29685 catalog_manager.cc:1347] Generated new cluster ID: c765740cb6214ebba574d645b4809659
I20260925 22:40:39.620663 29685 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:40:39.624279 29664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44f7db4fbd52462ca1f414b9e7f14212 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "44f7db4fbd52462ca1f414b9e7f14212" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f7db4fbd52462ca1f414b9e7f14212" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 33595 } } peers { permanent_uuid: "e25b5cc5f6304ff2aa60566cd7d60250" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 36333 } } peers { permanent_uuid: "3995b749ef0a472ea9cda13089d78126" member_type: VOTER last_known_addr { host: "127.6.246.252" port: 35781 } } }
I20260925 22:40:39.624384 29664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44f7db4fbd52462ca1f414b9e7f14212 [sys.catalog]: This master's current role is: LEADER
I20260925 22:40:39.625509 29650 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3995b749ef0a472ea9cda13089d78126 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "44f7db4fbd52462ca1f414b9e7f14212" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f7db4fbd52462ca1f414b9e7f14212" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 33595 } } peers { permanent_uuid: "e25b5cc5f6304ff2aa60566cd7d60250" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 36333 } } peers { permanent_uuid: "3995b749ef0a472ea9cda13089d78126" member_type: VOTER last_known_addr { host: "127.6.246.252" port: 35781 } } }
I20260925 22:40:39.625625 29650 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3995b749ef0a472ea9cda13089d78126 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:40:39.626111 29690 mvcc.cc:205] Tried to move back new op lower bound from 7333380258201890816 to 7333380258123960320. Current Snapshot: MvccSnapshot[applied={T|T < 7333380258201890816}]
I20260925 22:40:39.638334 29685 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:40:39.639595 29685 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:40:39.655412 29685 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 44f7db4fbd52462ca1f414b9e7f14212: Generated new TSK 0
I20260925 22:40:39.656348 29685 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 22:40:39.690877 29632 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:40:39.689353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46896 (local address 127.6.246.252:35781)
0925 22:40:39.689483 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:39.689488 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:39.689947 (+ 459us) server_negotiation.cc:407] Connection header received
0925 22:40:39.690128 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:39.690134 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:39.690183 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:39.690238 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:39.690728 (+ 490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46896: BlockingRecv error: recv got EOF from 127.0.0.1:46896 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260925 22:40:39.690877 29554 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:40:39.689704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36528 (local address 127.6.246.253:36333)
0925 22:40:39.689819 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:39.689825 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:39.689844 (+ 19us) server_negotiation.cc:407] Connection header received
0925 22:40:39.689900 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:39.689906 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:39.689966 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:39.690047 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:39.690764 (+ 717us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36528: BlockingRecv error: recv got EOF from 127.0.0.1:36528 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260925 22:40:39.691021 29482 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:40:39.689300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37950 (local address 127.6.246.254:33595)
0925 22:40:39.689441 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:39.689447 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:39.690060 (+ 613us) server_negotiation.cc:407] Connection header received
0925 22:40:39.690232 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:39.690235 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:39.690297 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:39.690353 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:39.690898 (+ 545us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37950: BlockingRecv error: recv got EOF from 127.0.0.1:37950 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260925 22:40:39.694308 29554 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:40:39.693192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36530 (local address 127.6.246.253:36333)
0925 22:40:39.693298 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:39.693301 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:39.693493 (+ 192us) server_negotiation.cc:407] Connection header received
0925 22:40:39.693676 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:39.693678 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:39.693749 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:39.693864 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:39.694215 (+ 351us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36530: BlockingRecv error: recv got EOF from 127.0.0.1:36530 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20260925 22:40:39.698053 29554 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:40:39.697279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36538 (local address 127.6.246.253:36333)
0925 22:40:39.697396 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:39.697399 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:39.697413 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:40:39.697454 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:39.697456 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:39.697491 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:39.697553 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:39.697961 (+ 408us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36538: BlockingRecv error: recv got EOF from 127.0.0.1:36538 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260925 22:40:39.702797 29554 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:40:39.701710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36546 (local address 127.6.246.253:36333)
0925 22:40:39.701836 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:39.701842 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:39.702047 (+ 205us) server_negotiation.cc:407] Connection header received
0925 22:40:39.702193 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:39.702196 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:39.702246 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:39.702340 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:39.702707 (+ 367us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36546: BlockingRecv error: recv got EOF from 127.0.0.1:36546 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260925 22:40:39.709613 29554 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:40:39.708562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36552 (local address 127.6.246.253:36333)
0925 22:40:39.708711 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:39.708715 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:39.708834 (+ 119us) server_negotiation.cc:407] Connection header received
0925 22:40:39.709005 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:39.709008 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:39.709052 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:39.709142 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:39.709521 (+ 379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36552: BlockingRecv error: recv got EOF from 127.0.0.1:36552 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
W20260925 22:40:39.720620 29554 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:40:39.719578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36560 (local address 127.6.246.253:36333)
0925 22:40:39.719705 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:39.719709 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:39.719829 (+ 120us) server_negotiation.cc:407] Connection header received
0925 22:40:39.719949 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:39.719952 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:39.720007 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:39.720107 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:39.720512 (+ 405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36560: BlockingRecv error: recv got EOF from 127.0.0.1:36560 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260925 22:40:39.740132 29554 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:40:39.738820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36564 (local address 127.6.246.253:36333)
0925 22:40:39.738962 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:39.738967 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:39.739086 (+ 119us) server_negotiation.cc:407] Connection header received
0925 22:40:39.739262 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:39.739266 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:39.739330 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:39.739448 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:39.740013 (+ 565us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36564: BlockingRecv error: recv got EOF from 127.0.0.1:36564 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260925 22:40:39.775578 29554 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:40:39.774390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36578 (local address 127.6.246.253:36333)
0925 22:40:39.774559 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:39.774563 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:39.774618 (+ 55us) server_negotiation.cc:407] Connection header received
0925 22:40:39.774799 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:39.774801 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:39.774848 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:39.774921 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:39.775423 (+ 502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36578: BlockingRecv error: recv got EOF from 127.0.0.1:36578 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
W20260925 22:40:39.843045 29554 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:40:39.841836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36592 (local address 127.6.246.253:36333)
0925 22:40:39.841992 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:39.841997 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:39.842119 (+ 122us) server_negotiation.cc:407] Connection header received
0925 22:40:39.842292 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:39.842296 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:39.842367 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:39.842460 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:39.842912 (+ 452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36592: BlockingRecv error: recv got EOF from 127.0.0.1:36592 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260925 22:40:39.974344 29554 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:40:39.973244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36604 (local address 127.6.246.253:36333)
0925 22:40:39.973388 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:39.973392 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:39.973517 (+ 125us) server_negotiation.cc:407] Connection header received
0925 22:40:39.973625 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:39.973627 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:39.973675 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:39.973768 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:39.974201 (+ 433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36604: BlockingRecv error: recv got EOF from 127.0.0.1:36604 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260925 22:40:40.234396 29554 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:40:40.233133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36608 (local address 127.6.246.253:36333)
0925 22:40:40.233296 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:40.233303 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:40.233325 (+ 22us) server_negotiation.cc:407] Connection header received
0925 22:40:40.233542 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:40.233545 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:40.233598 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:40.233692 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:40.234230 (+ 538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36608: BlockingRecv error: recv got EOF from 127.0.0.1:36608 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 22:40:40.428915 29647 catalog_manager.cc:1259] Loaded cluster ID: c765740cb6214ebba574d645b4809659
I20260925 22:40:40.429042 29647 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e25b5cc5f6304ff2aa60566cd7d60250: loading cluster ID for follower catalog manager: success
I20260925 22:40:40.430876 29647 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e25b5cc5f6304ff2aa60566cd7d60250: acquiring CA information for follower catalog manager: success
I20260925 22:40:40.432168 29647 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e25b5cc5f6304ff2aa60566cd7d60250: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 22:40:40.451059 29661 catalog_manager.cc:1259] Loaded cluster ID: c765740cb6214ebba574d645b4809659
I20260925 22:40:40.451167 29661 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3995b749ef0a472ea9cda13089d78126: loading cluster ID for follower catalog manager: success
I20260925 22:40:40.453291 29661 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3995b749ef0a472ea9cda13089d78126: acquiring CA information for follower catalog manager: success
I20260925 22:40:40.454631 29661 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3995b749ef0a472ea9cda13089d78126: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 22:40:40.753654 29761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:40.749121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36610 (local address 127.6.246.253:36333)
0925 22:40:40.749423 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:40.749429 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:40.749447 (+ 18us) server_negotiation.cc:407] Connection header received
0925 22:40:40.749644 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:40.749648 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:40.749813 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:40.749954 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:40:40.750356 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:40.751009 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:40.751953 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:40.752206 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:40.752293 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:40.752655 (+ 362us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:40:40.753230 (+ 575us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:40:40.753335 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:40.753419 (+ 84us) server_negotiation.cc:299] Negotiation successful
0925 22:40:40.753481 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":82,"threads_started":1}
I20260925 22:40:40.761483 29767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:40.757243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46902 (local address 127.6.246.252:35781)
0925 22:40:40.757610 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:40.757614 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:40.757631 (+ 17us) server_negotiation.cc:407] Connection header received
0925 22:40:40.757736 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:40.757738 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:40.757886 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:40.758044 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:40:40.758230 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:40.758802 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:40.759637 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:40.759896 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:40.759995 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:40.760367 (+ 372us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:40:40.760994 (+ 627us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:40:40.761115 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:40.761178 (+ 63us) server_negotiation.cc:299] Negotiation successful
0925 22:40:40.761237 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":114,"threads_started":1}
I20260925 22:40:40.769636 29761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:40.765083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36620 (local address 127.6.246.253:36333)
0925 22:40:40.765228 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:40.765233 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:40.765371 (+ 138us) server_negotiation.cc:407] Connection header received
0925 22:40:40.765596 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:40.765599 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:40.765658 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:40.765728 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:40:40.766240 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:40.767045 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:40.768291 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:40.768686 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:40.768796 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:40.769107 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:40:40.769190 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:40:40.769256 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:40.769434 (+ 178us) server_negotiation.cc:299] Negotiation successful
0925 22:40:40.769493 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 22:40:40.769647 29767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:40.764944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46912 (local address 127.6.246.252:35781)
0925 22:40:40.765075 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:40.765079 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:40.765477 (+ 398us) server_negotiation.cc:407] Connection header received
0925 22:40:40.765583 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:40.765586 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:40.765634 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:40.765711 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:40:40.766013 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:40.766842 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:40.768247 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:40.768563 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:40.768624 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:40.769003 (+ 379us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:40:40.769092 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:40:40.769129 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:40.769276 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 22:40:40.769342 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 22:40:40.771265 29774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:40.764794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37958 (local address 127.6.246.254:33595)
0925 22:40:40.765114 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:40.765119 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:40.765225 (+ 106us) server_negotiation.cc:407] Connection header received
0925 22:40:40.765467 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:40.765471 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:40.765681 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:40.765816 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:40:40.766179 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:40.766809 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:40.769398 (+ 2589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:40.769751 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:40.769849 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:40.770293 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:40:40.770771 (+ 478us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:40:40.770868 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:40.770972 (+ 104us) server_negotiation.cc:299] Negotiation successful
0925 22:40:40.771063 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":112,"threads_started":1}
I20260925 22:40:40.774554 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 29411
I20260925 22:40:40.782243 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 29486
I20260925 22:40:40.789450 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 29561
2026-09-25T22:40:40Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3288 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790375752797504-7131-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:40:40 dist-test-slave-wjdr krb5kdc[29778](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:40:40 dist-test-slave-wjdr krb5kdc[29778](info): set up 2 sockets
Sep 25 22:40:40 dist-test-slave-wjdr krb5kdc[29778](info): commencing operation
krb5kdc: starting...
W20260925 22:40:41.833868 7131 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790375752797504-7131-0/krb5kdc/test-user.keytab.
Sep 25 22:40:41 dist-test-slave-wjdr krb5kdc[29778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:40:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:40:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.246.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab.
Entry for principal kudu/127.6.246.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab.
I20260925 22:40:41.953946 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790375752797504-7131-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790375752797504-7131-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790375752797504-7131-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:33477
--webserver_interface=127.6.246.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab
--principal=kudu/127.6.246.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:35301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.246.254:33477,127.6.246.253:37165,127.6.246.252:46633
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:40:42.068435 29791 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 22:40:42.068816 29791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:40:42.068882 29791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:40:42.068979 29791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:40:42.072981 29791 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:40:42.073050 29791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:40:42.073098 29791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:40:42.073154 29791 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:40:42.073191 29791 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:40:42.078975 29791 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:35301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790375752797504-7131-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790375752797504-7131-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.246.254:33477,127.6.246.253:37165,127.6.246.252:46633
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.254.keytab
--principal=kudu/127.6.246.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.254:33477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.246.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790375752797504-7131-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:40:42.080024 29791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:40:42.081070 29791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:40:42.089414 29799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:42.089426 29796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:42.089429 29797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:40:42.090106 29791 server_base.cc:1034] running on GCE node
I20260925 22:40:42.090651 29791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:40:42.091837 29791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:40:42.093046 29791 hybrid_clock.cc:648] HybridClock initialized: now 1790376042093017 us; error 44 us; skew 500 ppm
Sep 25 22:40:42 dist-test-slave-wjdr krb5kdc[29778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376042, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:40:42.095932 29791 init.cc:377] Logged in from keytab as kudu/127.6.246.254@KRBTEST.COM (short username kudu)
I20260925 22:40:42.096988 29791 webserver.cc:460] Webserver started at http://127.6.246.254:46177/ using document root <none> and password file <none>
I20260925 22:40:42.097563 29791 fs_manager.cc:359] Metadata directory not provided
I20260925 22:40:42.097653 29791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:40:42.097873 29791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:40:42.099901 29791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790375752797504-7131-0/minicluster-data/master-0/data/instance:
uuid: "e0ef79cd85ab4015aa59b6f1e1cd25c3"
format_stamp: "Formatted at 2026-09-25 22:40:42 on dist-test-slave-wjdr"
server_key: "c33317f8ee1c908642189b08bb194f91"
server_key_iv: "3e76536d11d17b1f11a60e100d7b4203"
server_key_version: "encryptionkey@0"
I20260925 22:40:42.100530 29791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790375752797504-7131-0/minicluster-data/master-0/wal/instance:
uuid: "e0ef79cd85ab4015aa59b6f1e1cd25c3"
format_stamp: "Formatted at 2026-09-25 22:40:42 on dist-test-slave-wjdr"
server_key: "c33317f8ee1c908642189b08bb194f91"
server_key_iv: "3e76536d11d17b1f11a60e100d7b4203"
server_key_version: "encryptionkey@0"
I20260925 22:40:42.104275 29791 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260925 22:40:42.106786 29806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:42.108008 29791 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:40:42.108150 29791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790375752797504-7131-0/minicluster-data/master-0/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790375752797504-7131-0/minicluster-data/master-0/wal
uuid: "e0ef79cd85ab4015aa59b6f1e1cd25c3"
format_stamp: "Formatted at 2026-09-25 22:40:42 on dist-test-slave-wjdr"
server_key: "c33317f8ee1c908642189b08bb194f91"
server_key_iv: "3e76536d11d17b1f11a60e100d7b4203"
server_key_version: "encryptionkey@0"
I20260925 22:40:42.108268 29791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790375752797504-7131-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790375752797504-7131-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790375752797504-7131-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:40:42.131304 29791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:40:42.134763 29791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:40:42.134986 29791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:40:42.142673 29791 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.254:33477
I20260925 22:40:42.142956 29858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.254:33477 every 8 connection(s)
I20260925 22:40:42.143765 29791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790375752797504-7131-0/minicluster-data/master-0/data/info.pb
I20260925 22:40:42.146806 29859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:40:42.150473 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 29791
I20260925 22:40:42.150573 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790375752797504-7131-0/minicluster-data/master-0/wal/instance
I20260925 22:40:42.151044 7131 external_mini_cluster.cc:1428] Setting key e9193dd2c436baac6832b122913365bb
I20260925 22:40:42.150799 29859 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.246.254" port: 33477 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.6.246.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.253@KRBTEST.COM" created.
Sep 25 22:40:42 dist-test-slave-wjdr krb5kdc[29778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376042, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.254@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
I20260925 22:40:42.168054 29863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:42.153192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51312 (local address 127.6.246.254:33477)
0925 22:40:42.153760 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:42.153784 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:42.153993 (+ 209us) server_negotiation.cc:407] Connection header received
0925 22:40:42.155002 (+ 1009us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:42.155010 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:42.155277 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:42.155480 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:42.156321 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:42.157844 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:42.158966 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:42.159559 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:42.159728 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:42.163085 (+ 3357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:42.163096 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:42.163111 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:42.163142 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:42.164942 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:42.165446 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:42.165451 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:42.165453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:42.165562 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:42.165890 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:42.165896 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:42.165898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:42.166272 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:42.166408 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:42.166778 (+ 370us) server_negotiation.cc:299] Negotiation successful
0925 22:40:42.167028 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":412,"thread_start_us":145,"threads_started":1}
I20260925 22:40:42.168054 29862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:42.153191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:33477 (local address 127.0.0.1:51312)
0925 22:40:42.153723 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:42.153777 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:42.154678 (+ 901us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:42.155503 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:42.155507 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:42.155909 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:40:42.156155 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:42.156167 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:42.157987 (+ 1820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:42.157990 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:42.158815 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:42.158827 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:42.159763 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:42.159767 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:42.159909 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:42.160523 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:40:42.160543 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:40:42.162901 (+ 2358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:40:42.165069 (+ 2168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:42.165079 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:42.165089 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:42.165334 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:42.165676 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:42.165679 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:42.165681 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:42.165798 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:42.166390 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:40:42.166397 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:40:42.166539 (+ 142us) client_negotiation.cc:769] Sending connection context
0925 22:40:42.166709 (+ 170us) client_negotiation.cc:240] Negotiation successful
0925 22:40:42.167077 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":142,"thread_start_us":133,"threads_started":1}
I20260925 22:40:42.170883 29859 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.246.253" port: 37165 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:40:42.171695 29862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:42.171410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.253:37165 (local address 127.0.0.1:49132)
0925 22:40:42.171560 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:42.171635 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.253:37165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260925 22:40:42.171885 29808 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.246.253:37165: connect: Connection refused (error 111)
I20260925 22:40:42.172757 29862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:42.172556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.253:37165 (local address 127.0.0.1:49144)
0925 22:40:42.172657 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:42.172703 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.253:37165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260925 22:40:42.172907 29859 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.246.253:37165: Network error: Client connection negotiation failed: client connection to 127.6.246.253:37165: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.253.keytab.
Entry for principal kudu/127.6.246.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.253.keytab.
I20260925 22:40:42.179853 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790375752797504-7131-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790375752797504-7131-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790375752797504-7131-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790375752797504-7131-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.253:37165
--webserver_interface=127.6.246.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.253.keytab
--principal=kudu/127.6.246.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:35301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.246.254:33477,127.6.246.253:37165,127.6.246.252:46633
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:40:42.222193 29859 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.246.253" port: 37165 } attempt: 1
I20260925 22:40:42.223176 29862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:42.222857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.253:37165 (local address 127.0.0.1:49160)
0925 22:40:42.223016 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:42.223091 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.253:37165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260925 22:40:42.223977 29862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:42.223752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.253:37165 (local address 127.0.0.1:49172)
0925 22:40:42.223873 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:42.223923 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.253:37165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 22:40:42.224143 29859 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.246.253:37165: Network error: Client connection negotiation failed: client connection to 127.6.246.253:37165: connect: Connection refused (error 111)
I20260925 22:40:42.292325 29859 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.246.253" port: 37165 } attempt: 2
W20260925 22:40:42.292263 29866 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 22:40:42.292683 29866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:40:42.292769 29866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:40:42.292850 29866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260925 22:40:42.293447 29862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:42.293148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.253:37165 (local address 127.0.0.1:49182)
0925 22:40:42.293304 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:42.293374 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.253:37165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260925 22:40:42.294162 29862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:42.293944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.253:37165 (local address 127.0.0.1:49184)
0925 22:40:42.294057 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:42.294102 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.253:37165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 22:40:42.294301 29859 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.246.253:37165: Network error: Client connection negotiation failed: client connection to 127.6.246.253:37165: connect: Connection refused (error 111)
W20260925 22:40:42.297375 29866 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:40:42.297454 29866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:40:42.297492 29866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:40:42.297547 29866 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:40:42.297587 29866 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:40:42.303069 29866 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:35301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790375752797504-7131-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790375752797504-7131-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.6.246.254:33477,127.6.246.253:37165,127.6.246.252:46633
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.253.keytab
--principal=kudu/127.6.246.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.253:37165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790375752797504-7131-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.246.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790375752797504-7131-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:40:42.304167 29866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:40:42.305323 29866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:40:42.313979 29874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:42.313974 29872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:42.313974 29871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:40:42.314335 29866 server_base.cc:1034] running on GCE node
I20260925 22:40:42.314949 29866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:40:42.316020 29866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:40:42.317201 29866 hybrid_clock.cc:648] HybridClock initialized: now 1790376042317168 us; error 45 us; skew 500 ppm
Sep 25 22:40:42 dist-test-slave-wjdr krb5kdc[29778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376042, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:40:42.320120 29866 init.cc:377] Logged in from keytab as kudu/127.6.246.253@KRBTEST.COM (short username kudu)
I20260925 22:40:42.321049 29866 webserver.cc:460] Webserver started at http://127.6.246.253:41597/ using document root <none> and password file <none>
I20260925 22:40:42.321596 29866 fs_manager.cc:359] Metadata directory not provided
I20260925 22:40:42.321686 29866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:40:42.321914 29866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:40:42.323901 29866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790375752797504-7131-0/minicluster-data/master-1/data/instance:
uuid: "0c238cc38fa349e5a9a3fb53e148acea"
format_stamp: "Formatted at 2026-09-25 22:40:42 on dist-test-slave-wjdr"
server_key: "0e3e8e78f024e1b4c84b705e3ef6fd63"
server_key_iv: "04fbad0d1705078c408086e4ec249677"
server_key_version: "encryptionkey@0"
I20260925 22:40:42.324532 29866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790375752797504-7131-0/minicluster-data/master-1/wal/instance:
uuid: "0c238cc38fa349e5a9a3fb53e148acea"
format_stamp: "Formatted at 2026-09-25 22:40:42 on dist-test-slave-wjdr"
server_key: "0e3e8e78f024e1b4c84b705e3ef6fd63"
server_key_iv: "04fbad0d1705078c408086e4ec249677"
server_key_version: "encryptionkey@0"
I20260925 22:40:42.328147 29866 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260925 22:40:42.330868 29881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:42.332064 29866 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 22:40:42.332221 29866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790375752797504-7131-0/minicluster-data/master-1/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790375752797504-7131-0/minicluster-data/master-1/wal
uuid: "0c238cc38fa349e5a9a3fb53e148acea"
format_stamp: "Formatted at 2026-09-25 22:40:42 on dist-test-slave-wjdr"
server_key: "0e3e8e78f024e1b4c84b705e3ef6fd63"
server_key_iv: "04fbad0d1705078c408086e4ec249677"
server_key_version: "encryptionkey@0"
I20260925 22:40:42.332338 29866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790375752797504-7131-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790375752797504-7131-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790375752797504-7131-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:40:42.346616 29866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:40:42.349828 29866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:40:42.350029 29866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:40:42.356738 29866 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.253:37165
I20260925 22:40:42.357014 29933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.253:37165 every 8 connection(s)
I20260925 22:40:42.357671 29866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790375752797504-7131-0/minicluster-data/master-1/data/info.pb
I20260925 22:40:42.360348 29934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:40:42.363909 29934 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.246.254" port: 33477 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:40:42.366297 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 29866
I20260925 22:40:42.366441 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790375752797504-7131-0/minicluster-data/master-1/wal/instance
I20260925 22:40:42.366752 7131 external_mini_cluster.cc:1428] Setting key 2414a452da0ecb9ee2615a7414dcd749
Sep 25 22:40:42 dist-test-slave-wjdr krb5kdc[29778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376042, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.253@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.246.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.246.252@KRBTEST.COM" created.
I20260925 22:40:42.378155 29863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:42.365833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51318 (local address 127.6.246.254:33477)
0925 22:40:42.365973 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:42.365977 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:42.366902 (+ 925us) server_negotiation.cc:407] Connection header received
0925 22:40:42.367866 (+ 964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:42.367870 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:42.367934 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:42.368043 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:42.369243 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:42.369770 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:42.371568 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:42.371801 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:42.371884 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:42.374741 (+ 2857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:42.374745 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:42.374748 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:42.374775 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:42.376204 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:42.376730 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:42.376733 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:42.376734 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:42.376808 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:42.377092 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:42.377094 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:42.377095 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:42.377258 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:42.377329 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:42.377845 (+ 516us) server_negotiation.cc:299] Negotiation successful
0925 22:40:42.377979 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 22:40:42.378937 29936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:42.366207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:33477 (local address 127.0.0.1:51318)
0925 22:40:42.366739 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:42.366781 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:42.367645 (+ 864us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:42.368211 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:42.368217 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:42.368639 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:40:42.369089 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:42.369101 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:42.369906 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:42.369913 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:42.371438 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:42.371453 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:42.371965 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:42.371973 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:42.372233 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:42.372897 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:40:42.372920 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:40:42.374561 (+ 1641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:40:42.376304 (+ 1743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:42.376308 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:42.376322 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:42.376641 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:42.376888 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:42.376890 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:42.376891 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:42.377028 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:42.377340 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:40:42.377344 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:40:42.377577 (+ 233us) client_negotiation.cc:769] Sending connection context
0925 22:40:42.377769 (+ 192us) client_negotiation.cc:240] Negotiation successful
0925 22:40:42.378105 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":143,"threads_started":1}
I20260925 22:40:42.380285 29934 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.246.253" port: 37165 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:40:42.385464 29859 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.246.253" port: 37165 } attempt: 3
Sep 25 22:40:42 dist-test-slave-wjdr krb5kdc[29778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376042, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.253@KRBTEST.COM for kudu/127.6.246.253@KRBTEST.COM
Sep 25 22:40:42 dist-test-slave-wjdr krb5kdc[29778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376042, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.254@KRBTEST.COM for kudu/127.6.246.253@KRBTEST.COM
I20260925 22:40:42.393079 29936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:42.380872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.253:37165 (local address 127.0.0.1:49200)
0925 22:40:42.380990 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:42.381006 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:42.381086 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:42.381709 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:42.381711 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:42.381909 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:40:42.381999 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:42.382004 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:42.383523 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:42.383525 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:42.384044 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:42.384051 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:42.384924 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:42.384926 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:42.385004 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:42.385476 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:40:42.385488 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:40:42.389000 (+ 3512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:40:42.391175 (+ 2175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:42.391179 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:42.391182 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:42.391424 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:42.391768 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:42.391771 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:42.391773 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:42.391849 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:42.392535 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:40:42.392538 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:40:42.392638 (+ 100us) client_negotiation.cc:769] Sending connection context
0925 22:40:42.392767 (+ 129us) client_negotiation.cc:240] Negotiation successful
0925 22:40:42.392906 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260925 22:40:42.393201 29939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:42.380949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49200 (local address 127.6.246.253:37165)
0925 22:40:42.381275 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:42.381289 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:42.381313 (+ 24us) server_negotiation.cc:407] Connection header received
0925 22:40:42.381368 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:42.381376 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:42.381574 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:42.381723 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:42.382114 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:42.383406 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:42.384157 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:42.384802 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:42.384932 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:42.389169 (+ 4237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:42.389177 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:42.389189 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:42.389208 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:42.391066 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:42.391561 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:42.391567 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:42.391569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:42.391678 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:42.391966 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:42.391969 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:42.391970 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:42.392324 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:42.392539 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:42.392898 (+ 359us) server_negotiation.cc:299] Negotiation successful
0925 22:40:42.393026 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":104,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.246.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.252.keytab.
Entry for principal kudu/127.6.246.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.252.keytab.
I20260925 22:40:42.395565 29934 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.246.252" port: 46633 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:40:42.395879 29862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:42.385926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.253:37165 (local address 127.0.0.1:49202)
0925 22:40:42.386149 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:42.386165 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:42.386237 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:42.386697 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:42.386699 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:42.386832 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:40:42.386921 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:42.386926 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:42.387635 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:42.387637 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:42.388985 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:42.388995 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:42.389468 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:42.389472 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:42.389562 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:42.389887 (+ 325us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:40:42.389902 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:40:42.391309 (+ 1407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:40:42.393920 (+ 2611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:42.393923 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:42.393924 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:42.394119 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:42.394485 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:42.394491 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:42.394494 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:42.394586 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:42.394997 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:40:42.395000 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:40:42.395106 (+ 106us) client_negotiation.cc:769] Sending connection context
0925 22:40:42.395239 (+ 133us) client_negotiation.cc:240] Negotiation successful
0925 22:40:42.395640 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132}
I20260925 22:40:42.395864 7131 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskan1x8t/build/debug/bin/kudu
/tmp/dist-test-taskan1x8t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790375752797504-7131-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790375752797504-7131-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790375752797504-7131-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790375752797504-7131-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.252:46633
--webserver_interface=127.6.246.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.252.keytab
--principal=kudu/127.6.246.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.246.212:35301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.246.254:33477,127.6.246.253:37165,127.6.246.252:46633
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790375752797504-7131-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790375752797504-7131-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790375752797504-7131-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:40:42.396522 29936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:42.396056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.252:46633 (local address 127.0.0.1:53180)
0925 22:40:42.396349 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:42.396415 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.252:46633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232}
I20260925 22:40:42.396682 29940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:42.385997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49202 (local address 127.6.246.253:37165)
0925 22:40:42.386502 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:42.386509 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:42.386521 (+ 12us) server_negotiation.cc:407] Connection header received
0925 22:40:42.386547 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:42.386549 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:42.386582 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:42.386652 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:42.387021 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:42.387542 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:42.389099 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:42.389354 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:42.389467 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:42.391444 (+ 1977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:42.391447 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:42.391449 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:42.391468 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:42.393811 (+ 2343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:42.394258 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:42.394261 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:42.394263 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:42.394354 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:42.394680 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:42.394682 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:42.394683 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:42.394882 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:42.395005 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:42.396205 (+ 1200us) server_negotiation.cc:299] Negotiation successful
0925 22:40:42.396338 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":156,"threads_started":1}
W20260925 22:40:42.396835 29884 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.246.252:46633: connect: Connection refused (error 111)
I20260925 22:40:42.397715 29859 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.246.252" port: 46633 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:40:42.397927 29936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:42.397653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.252:46633 (local address 127.0.0.1:53194)
0925 22:40:42.397809 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:42.397859 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.252:46633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260925 22:40:42.398046 29934 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.246.252:46633: Network error: Client connection negotiation failed: client connection to 127.6.246.252:46633: connect: Connection refused (error 111)
I20260925 22:40:42.398403 29862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:42.398198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.252:46633 (local address 127.0.0.1:53206)
0925 22:40:42.398290 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:42.398338 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.252:46633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260925 22:40:42.399036 29862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:42.398791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.252:46633 (local address 127.0.0.1:53210)
0925 22:40:42.398935 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:42.398986 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.252:46633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260925 22:40:42.399175 29859 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.246.252:46633: Network error: Client connection negotiation failed: client connection to 127.6.246.252:46633: connect: Connection refused (error 111)
I20260925 22:40:42.430435 29859 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.246.252" port: 46633 } attempt: 1
I20260925 22:40:42.431344 29862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:42.431046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.252:46633 (local address 127.0.0.1:53222)
0925 22:40:42.431210 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:42.431271 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.252:46633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260925 22:40:42.432036 29862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:42.431825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.252:46633 (local address 127.0.0.1:53232)
0925 22:40:42.431937 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:42.431986 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.252:46633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260925 22:40:42.432159 29859 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.246.252:46633: Network error: Client connection negotiation failed: client connection to 127.6.246.252:46633: connect: Connection refused (error 111)
I20260925 22:40:42.447346 29934 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.246.252" port: 46633 } attempt: 1
I20260925 22:40:42.448207 29936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:42.447927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.252:46633 (local address 127.0.0.1:53234)
0925 22:40:42.448077 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:42.448136 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.252:46633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260925 22:40:42.448956 29936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:42.448745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.252:46633 (local address 127.0.0.1:53238)
0925 22:40:42.448854 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:42.448904 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.252:46633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 22:40:42.449076 29934 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.246.252:46633: Network error: Client connection negotiation failed: client connection to 127.6.246.252:46633: connect: Connection refused (error 111)
I20260925 22:40:42.507344 29859 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.246.252" port: 46633 } attempt: 2
I20260925 22:40:42.508217 29862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:42.507924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.252:46633 (local address 127.0.0.1:53254)
0925 22:40:42.508082 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:42.508144 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.252:46633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260925 22:40:42.508575 29942 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 22:40:42.508888 29942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:40:42.508958 29942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
I20260925 22:40:42.508950 29862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:42.508733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.252:46633 (local address 127.0.0.1:53268)
0925 22:40:42.508838 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:42.508888 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.252:46633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 22:40:42.509035 29942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:40:42.509112 29859 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.246.252:46633: Network error: Client connection negotiation failed: client connection to 127.6.246.252:46633: connect: Connection refused (error 111)
W20260925 22:40:42.513231 29942 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:40:42.513305 29942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:40:42.513350 29942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:40:42.513386 29942 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:40:42.513420 29942 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:40:42.517232 29934 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.246.252" port: 46633 } attempt: 2
I20260925 22:40:42.517942 29936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:42.517689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.252:46633 (local address 127.0.0.1:53270)
0925 22:40:42.517816 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:42.517878 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.252:46633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260925 22:40:42.518608 29936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:42.518402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.252:46633 (local address 127.0.0.1:53278)
0925 22:40:42.518507 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:42.518551 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.246.252:46633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260925 22:40:42.518718 29934 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.246.252:46633: Network error: Client connection negotiation failed: client connection to 127.6.246.252:46633: connect: Connection refused (error 111)
I20260925 22:40:42.518994 29942 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.246.212:35301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790375752797504-7131-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790375752797504-7131-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.6.246.254:33477,127.6.246.253:37165,127.6.246.252:46633
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790375752797504-7131-0/krb5kdc/kudu_127.6.246.252.keytab
--principal=kudu/127.6.246.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.246.252:46633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790375752797504-7131-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.246.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29942
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790375752797504-7131-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:00:54 UTC on ccycloud.quasar-apiybq.root.comops.site
build id 4673383
I20260925 22:40:42.520064 29942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:40:42.521121 29942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:40:42.529392 29948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:40:42.529582 29950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:40:42.529577 29942 server_base.cc:1034] running on GCE node
W20260925 22:40:42.529433 29947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:40:42.530251 29942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:40:42.531322 29942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:40:42.532531 29942 hybrid_clock.cc:648] HybridClock initialized: now 1790376042532519 us; error 30 us; skew 500 ppm
Sep 25 22:40:42 dist-test-slave-wjdr krb5kdc[29778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376042, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:40:42.539022 29942 init.cc:377] Logged in from keytab as kudu/127.6.246.252@KRBTEST.COM (short username kudu)
I20260925 22:40:42.539877 29942 webserver.cc:460] Webserver started at http://127.6.246.252:46341/ using document root <none> and password file <none>
I20260925 22:40:42.540388 29942 fs_manager.cc:359] Metadata directory not provided
I20260925 22:40:42.540503 29942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:40:42.540733 29942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:40:42.542802 29942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790375752797504-7131-0/minicluster-data/master-2/data/instance:
uuid: "712c0ff960f04f06a6941d607ecdb2a1"
format_stamp: "Formatted at 2026-09-25 22:40:42 on dist-test-slave-wjdr"
server_key: "e0d4e31f248e41488d1aacf37e5f9ebe"
server_key_iv: "4e1d28996e7c0c72cd79aea607f4993e"
server_key_version: "encryptionkey@0"
I20260925 22:40:42.543357 29942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790375752797504-7131-0/minicluster-data/master-2/wal/instance:
uuid: "712c0ff960f04f06a6941d607ecdb2a1"
format_stamp: "Formatted at 2026-09-25 22:40:42 on dist-test-slave-wjdr"
server_key: "e0d4e31f248e41488d1aacf37e5f9ebe"
server_key_iv: "4e1d28996e7c0c72cd79aea607f4993e"
server_key_version: "encryptionkey@0"
I20260925 22:40:42.546929 29942 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260925 22:40:42.549204 29957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:40:42.550407 29942 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 22:40:42.550541 29942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790375752797504-7131-0/minicluster-data/master-2/data,/tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790375752797504-7131-0/minicluster-data/master-2/wal
uuid: "712c0ff960f04f06a6941d607ecdb2a1"
format_stamp: "Formatted at 2026-09-25 22:40:42 on dist-test-slave-wjdr"
server_key: "e0d4e31f248e41488d1aacf37e5f9ebe"
server_key_iv: "4e1d28996e7c0c72cd79aea607f4993e"
server_key_version: "encryptionkey@0"
I20260925 22:40:42.550643 29942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790375752797504-7131-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790375752797504-7131-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790375752797504-7131-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:40:42.567265 29942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:40:42.570214 29942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:40:42.570425 29942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:40:42.577494 29942 rpc_server.cc:307] RPC server started. Bound to: 127.6.246.252:46633
I20260925 22:40:42.577816 30009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.246.252:46633 every 8 connection(s)
I20260925 22:40:42.578476 29942 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790375752797504-7131-0/minicluster-data/master-2/data/info.pb
I20260925 22:40:42.581283 30010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:40:42.582804 7131 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskan1x8t/build/debug/bin/kudu as pid 29942
I20260925 22:40:42.582914 7131 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskan1x8t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790375752797504-7131-0/minicluster-data/master-2/wal/instance
I20260925 22:40:42.583220 7131 external_mini_cluster.cc:1428] Setting key cafec9350ea46b62a73086d95475b494
I20260925 22:40:42.585343 30010 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.246.254" port: 33477 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 22:40:42 dist-test-slave-wjdr krb5kdc[29778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.246.252@KRBTEST.COM
Sep 25 22:40:42 dist-test-slave-wjdr krb5kdc[29778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376042, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.252@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
I20260925 22:40:42.597487 29863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:42.587686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51332 (local address 127.6.246.254:33477)
0925 22:40:42.587864 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:42.587868 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:42.588160 (+ 292us) server_negotiation.cc:407] Connection header received
0925 22:40:42.588330 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:42.588335 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:42.588396 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:42.588503 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:42.589348 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:42.589884 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:42.591049 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:42.591261 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:42.591339 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:42.593904 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:42.593910 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:42.593914 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:42.593943 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:42.595447 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:42.595913 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:42.595918 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:42.595919 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:42.596006 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:42.596386 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:42.596389 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:42.596391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:42.596665 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:42.596822 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:42.597184 (+ 362us) server_negotiation.cc:299] Negotiation successful
0925 22:40:42.597310 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260925 22:40:42.598305 30013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:42.584610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53280 (local address 127.6.246.252:46633)
0925 22:40:42.584996 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:42.585011 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:42.585044 (+ 33us) server_negotiation.cc:407] Connection header received
0925 22:40:42.585674 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:42.585690 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:42.586010 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:42.586320 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:42.587246 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:42.588668 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:42.589750 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:42.590330 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:42.590457 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:42.593337 (+ 2880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:42.593346 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:42.593363 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:42.593390 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:42.595410 (+ 2020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:42.596138 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:42.596146 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:42.596151 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:42.596257 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:42.596576 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:42.596580 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:42.596582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:42.596950 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:42.597177 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:42.597395 (+ 218us) server_negotiation.cc:299] Negotiation successful
0925 22:40:42.597537 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":249,"thread_start_us":124,"threads_started":1}
I20260925 22:40:42.598304 30014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:42.587586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.254:33477 (local address 127.0.0.1:51332)
0925 22:40:42.588038 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:42.588077 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:42.588221 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:42.588550 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:42.588555 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:42.589067 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:40:42.589223 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:42.589232 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:42.590010 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:42.590013 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:42.590897 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:42.590906 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:42.591420 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:42.591425 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:42.591565 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:42.592080 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:40:42.592107 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:40:42.593746 (+ 1639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:40:42.595564 (+ 1818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:42.595571 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:42.595580 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:42.595803 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:42.596104 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:42.596108 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:42.596111 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:42.596269 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:42.596817 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:40:42.596823 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:40:42.596968 (+ 145us) client_negotiation.cc:769] Sending connection context
0925 22:40:42.597141 (+ 173us) client_negotiation.cc:240] Negotiation successful
0925 22:40:42.597456 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"mutex_wait_us":49,"thread_start_us":122,"threads_started":1}
I20260925 22:40:42.599619 30010 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.246.253" port: 37165 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 22:40:42 dist-test-slave-wjdr krb5kdc[29778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376042, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.252@KRBTEST.COM for kudu/127.6.246.253@KRBTEST.COM
Sep 25 22:40:42 dist-test-slave-wjdr krb5kdc[29778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.246.254@KRBTEST.COM
I20260925 22:40:42.608268 29859 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.246.252" port: 46633 } attempt: 3
I20260925 22:40:42.608687 29940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:42.600353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49210 (local address 127.6.246.253:37165)
0925 22:40:42.600543 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:42.600549 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:42.600566 (+ 17us) server_negotiation.cc:407] Connection header received
0925 22:40:42.600611 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:42.600615 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:42.600661 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:42.600736 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:42.601205 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:42.601929 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:42.602822 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:42.603183 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:42.603301 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:42.605364 (+ 2063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:42.605367 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:42.605369 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:42.605391 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:42.606915 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:42.607447 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:42.607449 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:42.607451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:42.607522 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:42.607818 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:42.607820 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:42.607821 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:42.608022 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:42.608123 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:42.608381 (+ 258us) server_negotiation.cc:299] Negotiation successful
0925 22:40:42.608529 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 22:40:42.608687 30014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:42.600249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.253:37165 (local address 127.0.0.1:49210)
0925 22:40:42.600422 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:42.600436 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:42.600526 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:42.600780 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:42.600783 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:42.600960 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:40:42.601064 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:42.601071 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:42.602032 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:42.602036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:42.602716 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:42.602724 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:42.603331 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:42.603335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:42.603422 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:42.603869 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:40:42.603885 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:40:42.605192 (+ 1307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:40:42.607042 (+ 1850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:42.607044 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:42.607046 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:42.607342 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:42.607620 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:42.607625 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:42.607629 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:42.607725 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:42.608120 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:40:42.608122 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:40:42.608196 (+ 74us) client_negotiation.cc:769] Sending connection context
0925 22:40:42.608306 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 22:40:42.608529 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105}
I20260925 22:40:42.609853 30010 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.246.252" port: 46633 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:40:42.610088 29863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:42.600672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51348 (local address 127.6.246.254:33477)
0925 22:40:42.600850 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:42.600854 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:42.600868 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:40:42.600902 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:42.600907 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:42.600950 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:42.601035 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:42.601756 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:42.602389 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:42.603319 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:42.603674 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:42.603808 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:42.606193 (+ 2385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:42.606196 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:42.606199 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:42.606249 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:42.607915 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:42.608421 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:42.608424 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:42.608426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:42.608536 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:42.608841 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:42.608844 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:42.608846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:42.609084 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:42.609305 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:42.609809 (+ 504us) server_negotiation.cc:299] Negotiation successful
0925 22:40:42.609913 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260925 22:40:42.610121 29934 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.246.252" port: 46633 } attempt: 3
Sep 25 22:40:42 dist-test-slave-wjdr krb5kdc[29778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376042, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.254@KRBTEST.COM for kudu/127.6.246.252@KRBTEST.COM
Sep 25 22:40:42 dist-test-slave-wjdr krb5kdc[29778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376042, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.253@KRBTEST.COM for kudu/127.6.246.252@KRBTEST.COM
Sep 25 22:40:42 dist-test-slave-wjdr krb5kdc[29778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376042, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.246.252@KRBTEST.COM for kudu/127.6.246.252@KRBTEST.COM
Sep 25 22:40:42 dist-test-slave-wjdr krb5kdc[29778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.246.253@KRBTEST.COM
I20260925 22:40:42.618404 29862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:42.608683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.252:46633 (local address 127.0.0.1:53284)
0925 22:40:42.608957 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:42.608971 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:42.609038 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:42.609424 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:42.609427 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:42.609592 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:40:42.609713 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:42.609719 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:42.610887 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:42.610891 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:42.611700 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:42.611713 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:42.612324 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:42.612328 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:42.612424 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:42.612927 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:40:42.612946 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:40:42.614393 (+ 1447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:40:42.616778 (+ 2385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:42.616782 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:42.616784 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:42.617039 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:42.617353 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:42.617358 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:42.617362 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:42.617450 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:42.617891 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:40:42.617894 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:40:42.617992 (+ 98us) client_negotiation.cc:769] Sending connection context
0925 22:40:42.618126 (+ 134us) client_negotiation.cc:240] Negotiation successful
0925 22:40:42.618270 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202}
I20260925 22:40:42.618515 30013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:42.608971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53284 (local address 127.6.246.252:46633)
0925 22:40:42.609181 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:42.609186 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:42.609198 (+ 12us) server_negotiation.cc:407] Connection header received
0925 22:40:42.609229 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:42.609232 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:42.609273 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:42.609408 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:42.609835 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:42.610742 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:42.611840 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:42.612185 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:42.612657 (+ 472us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:42.614533 (+ 1876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:42.614537 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:42.614539 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:42.614563 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:42.616649 (+ 2086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:42.617157 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:42.617163 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:42.617167 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:42.617256 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:42.617562 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:42.617567 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:42.617570 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:42.617771 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:42.617895 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:42.618204 (+ 309us) server_negotiation.cc:299] Negotiation successful
0925 22:40:42.618319 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139}
I20260925 22:40:42.622138 30014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:42.610291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.252:46633 (local address 127.0.0.1:53298)
0925 22:40:42.610398 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:42.610412 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:42.610479 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:42.610984 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:42.610987 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:42.611165 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:40:42.611252 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:42.611259 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:42.613046 (+ 1787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:42.613050 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:42.613801 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:42.613810 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:42.614400 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:42.614403 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:42.614473 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:42.615108 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:40:42.615127 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:40:42.617620 (+ 2493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:40:42.620449 (+ 2829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:42.620506 (+ 57us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:42.620511 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:42.620761 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:42.621054 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:42.621057 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:42.621062 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:42.621157 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:42.621611 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:40:42.621614 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:40:42.621714 (+ 100us) client_negotiation.cc:769] Sending connection context
0925 22:40:42.621835 (+ 121us) client_negotiation.cc:240] Negotiation successful
0925 22:40:42.621969 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"spinlock_wait_cycles":896}
I20260925 22:40:42.622167 29859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0ef79cd85ab4015aa59b6f1e1cd25c3: Bootstrap starting.
I20260925 22:40:42.622182 30015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:42.610437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53298 (local address 127.6.246.252:46633)
0925 22:40:42.610745 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:42.610752 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:42.610764 (+ 12us) server_negotiation.cc:407] Connection header received
0925 22:40:42.610793 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:42.610795 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:42.610831 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:42.610946 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:42.611366 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:42.612943 (+ 1577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:42.613931 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:42.614296 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:42.615725 (+ 1429us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:42.617757 (+ 2032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:42.617760 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:42.617764 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:42.617794 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:42.620096 (+ 2302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:42.620854 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:42.620857 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:42.620859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:42.620948 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:42.621256 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:42.621261 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:42.621263 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:42.621518 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:42.621617 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:42.621887 (+ 270us) server_negotiation.cc:299] Negotiation successful
0925 22:40:42.622026 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":251,"thread_start_us":119,"threads_started":1}
I20260925 22:40:42.623106 29940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:42.611914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49212 (local address 127.6.246.253:37165)
0925 22:40:42.612002 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:42.612005 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:42.612257 (+ 252us) server_negotiation.cc:407] Connection header received
0925 22:40:42.612295 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:42.612297 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:42.612336 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:42.612412 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:42.613299 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:42.614092 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:42.615213 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:42.615578 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:42.615687 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:42.619481 (+ 3794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:42.619483 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:42.619485 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:42.619505 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:42.621109 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:42.621542 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:42.621546 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:42.621548 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:42.621627 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:42.621924 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:42.621927 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:42.621929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:42.622126 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:42.622322 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:42.622692 (+ 370us) server_negotiation.cc:299] Negotiation successful
0925 22:40:42.622937 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 22:40:42.625108 29859 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0ef79cd85ab4015aa59b6f1e1cd25c3: Neither blocks nor log segments found. Creating new log.
I20260925 22:40:42.625391 30010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 712c0ff960f04f06a6941d607ecdb2a1: Bootstrap starting.
I20260925 22:40:42.626052 29859 log.cc:824] T 00000000000000000000000000000000 P e0ef79cd85ab4015aa59b6f1e1cd25c3: Log is configured to *not* fsync() on all Append() calls
I20260925 22:40:42.626398 7131 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:40:42.627703 30010 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 712c0ff960f04f06a6941d607ecdb2a1: Neither blocks nor log segments found. Creating new log.
I20260925 22:40:42.628545 30010 log.cc:824] T 00000000000000000000000000000000 P 712c0ff960f04f06a6941d607ecdb2a1: Log is configured to *not* fsync() on all Append() calls
I20260925 22:40:42.627807 29833 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "e0ef79cd85ab4015aa59b6f1e1cd25c3"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51348
I20260925 22:40:42.629326 29859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0ef79cd85ab4015aa59b6f1e1cd25c3: No bootstrap required, opened a new log
I20260925 22:40:42.629638 29936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:42.610487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.246.252:46633 (local address 127.0.0.1:53300)
0925 22:40:42.610883 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0925 22:40:42.610896 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:40:42.610989 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:40:42.611997 (+ 1008us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:40:42.612000 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:40:42.612166 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:40:42.612252 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:42.612279 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:42.613449 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:42.613452 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:42.614236 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:40:42.614246 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:40:42.614898 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:40:42.614903 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:40:42.615003 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:42.615402 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:40:42.615425 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:40:42.618144 (+ 2719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:40:42.627903 (+ 9759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:42.627909 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:42.627913 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:42.628166 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:42.628529 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:40:42.628532 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:40:42.628534 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:40:42.628629 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:40:42.629127 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:40:42.629130 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:40:42.629238 (+ 108us) client_negotiation.cc:769] Sending connection context
0925 22:40:42.629340 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 22:40:42.629489 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316}
I20260925 22:40:42.630247 30016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:42.611186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53300 (local address 127.6.246.252:46633)
0925 22:40:42.611676 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:42.611690 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:42.611703 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:40:42.611735 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:42.611738 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:42.611787 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:42.612405 (+ 618us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:42.612568 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:42.613320 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:42.614373 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:42.614763 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:42.615218 (+ 455us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:42.618731 (+ 3513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:42.618735 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:42.618739 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:42.618770 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:42.627760 (+ 8990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:42.628289 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:42.628293 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:42.628295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:42.628388 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:42.628762 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:42.628765 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:42.628766 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:42.629004 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:42.629900 (+ 896us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:42.629974 (+ 74us) server_negotiation.cc:299] Negotiation successful
0925 22:40:42.630086 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":80,"threads_started":1}
I20260925 22:40:42.630517 30010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 712c0ff960f04f06a6941d607ecdb2a1: No bootstrap required, opened a new log
I20260925 22:40:42.632436 29859 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0ef79cd85ab4015aa59b6f1e1cd25c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0ef79cd85ab4015aa59b6f1e1cd25c3" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 33477 } } peers { permanent_uuid: "0c238cc38fa349e5a9a3fb53e148acea" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 37165 } } peers { permanent_uuid: "712c0ff960f04f06a6941d607ecdb2a1" member_type: VOTER last_known_addr { host: "127.6.246.252" port: 46633 } }
I20260925 22:40:42.632700 29859 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0ef79cd85ab4015aa59b6f1e1cd25c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0ef79cd85ab4015aa59b6f1e1cd25c3, State: Initialized, Role: FOLLOWER
I20260925 22:40:42.633020 29859 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0ef79cd85ab4015aa59b6f1e1cd25c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0ef79cd85ab4015aa59b6f1e1cd25c3" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 33477 } } peers { permanent_uuid: "0c238cc38fa349e5a9a3fb53e148acea" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 37165 } } peers { permanent_uuid: "712c0ff960f04f06a6941d607ecdb2a1" member_type: VOTER last_known_addr { host: "127.6.246.252" port: 46633 } }
I20260925 22:40:42.633138 29934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c238cc38fa349e5a9a3fb53e148acea: Bootstrap starting.
I20260925 22:40:42.633531 30019 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0ef79cd85ab4015aa59b6f1e1cd25c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0ef79cd85ab4015aa59b6f1e1cd25c3" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 33477 } } peers { permanent_uuid: "0c238cc38fa349e5a9a3fb53e148acea" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 37165 } } peers { permanent_uuid: "712c0ff960f04f06a6941d607ecdb2a1" member_type: VOTER last_known_addr { host: "127.6.246.252" port: 46633 } } }
I20260925 22:40:42.633658 30019 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0ef79cd85ab4015aa59b6f1e1cd25c3 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:40:42.633939 30010 raft_consensus.cc:348] T 00000000000000000000000000000000 P 712c0ff960f04f06a6941d607ecdb2a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0ef79cd85ab4015aa59b6f1e1cd25c3" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 33477 } } peers { permanent_uuid: "0c238cc38fa349e5a9a3fb53e148acea" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 37165 } } peers { permanent_uuid: "712c0ff960f04f06a6941d607ecdb2a1" member_type: VOTER last_known_addr { host: "127.6.246.252" port: 46633 } }
I20260925 22:40:42.634099 30010 raft_consensus.cc:729] T 00000000000000000000000000000000 P 712c0ff960f04f06a6941d607ecdb2a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 712c0ff960f04f06a6941d607ecdb2a1, State: Initialized, Role: FOLLOWER
I20260925 22:40:42.634307 30010 consensus_queue.cc:260] T 00000000000000000000000000000000 P 712c0ff960f04f06a6941d607ecdb2a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0ef79cd85ab4015aa59b6f1e1cd25c3" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 33477 } } peers { permanent_uuid: "0c238cc38fa349e5a9a3fb53e148acea" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 37165 } } peers { permanent_uuid: "712c0ff960f04f06a6941d607ecdb2a1" member_type: VOTER last_known_addr { host: "127.6.246.252" port: 46633 } }
I20260925 22:40:42.634370 29859 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0ef79cd85ab4015aa59b6f1e1cd25c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:40:42.634675 30020 sys_catalog.cc:455] T 00000000000000000000000000000000 P 712c0ff960f04f06a6941d607ecdb2a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0ef79cd85ab4015aa59b6f1e1cd25c3" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 33477 } } peers { permanent_uuid: "0c238cc38fa349e5a9a3fb53e148acea" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 37165 } } peers { permanent_uuid: "712c0ff960f04f06a6941d607ecdb2a1" member_type: VOTER last_known_addr { host: "127.6.246.252" port: 46633 } } }
I20260925 22:40:42.634788 30020 sys_catalog.cc:458] T 00000000000000000000000000000000 P 712c0ff960f04f06a6941d607ecdb2a1 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:40:42.635412 30010 sys_catalog.cc:564] T 00000000000000000000000000000000 P 712c0ff960f04f06a6941d607ecdb2a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:40:42.635919 29833 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0ef79cd85ab4015aa59b6f1e1cd25c3 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 22:40:42.636040 29833 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0ef79cd85ab4015aa59b6f1e1cd25c3 [term 0 FOLLOWER]: Advancing to term 1
W20260925 22:40:42.636582 29812 tablet.cc:2369] T 00000000000000000000000000000000 P e0ef79cd85ab4015aa59b6f1e1cd25c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:40:42.637126 29934 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c238cc38fa349e5a9a3fb53e148acea: Neither blocks nor log segments found. Creating new log.
I20260925 22:40:42.637718 29833 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0ef79cd85ab4015aa59b6f1e1cd25c3 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0ef79cd85ab4015aa59b6f1e1cd25c3" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 33477 } } peers { permanent_uuid: "0c238cc38fa349e5a9a3fb53e148acea" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 37165 } } peers { permanent_uuid: "712c0ff960f04f06a6941d607ecdb2a1" member_type: VOTER last_known_addr { host: "127.6.246.252" port: 46633 } }
I20260925 22:40:42.638262 29934 log.cc:824] T 00000000000000000000000000000000 P 0c238cc38fa349e5a9a3fb53e148acea: Log is configured to *not* fsync() on all Append() calls
I20260925 22:40:42.639216 29984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e0ef79cd85ab4015aa59b6f1e1cd25c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "712c0ff960f04f06a6941d607ecdb2a1"
I20260925 22:40:42.639446 29984 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 712c0ff960f04f06a6941d607ecdb2a1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:40:42.640628 29833 leader_election.cc:288] T 00000000000000000000000000000000 P e0ef79cd85ab4015aa59b6f1e1cd25c3 [CANDIDATE]: Term 1 election: Requested vote from peers 0c238cc38fa349e5a9a3fb53e148acea (127.6.246.253:37165), 712c0ff960f04f06a6941d607ecdb2a1 (127.6.246.252:46633)
I20260925 22:40:42.640995 29984 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 712c0ff960f04f06a6941d607ecdb2a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0ef79cd85ab4015aa59b6f1e1cd25c3 in term 1.
I20260925 22:40:42.639165 29942 master_runner.cc:186] Error getting master registration for 127.6.246.254:33477: OK, instance_id { permanent_uuid: "e0ef79cd85ab4015aa59b6f1e1cd25c3" instance_seqno: 1790376042140347 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 22:40:42.642660 29942 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 22:40:42.642750 29934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c238cc38fa349e5a9a3fb53e148acea: No bootstrap required, opened a new log
W20260925 22:40:42.643674 30041 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 712c0ff960f04f06a6941d607ecdb2a1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:40:42.643733 30041 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260925 22:40:42.644517 30042 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e0ef79cd85ab4015aa59b6f1e1cd25c3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:40:42.644585 30042 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:40:42.644672 29809 leader_election.cc:302] T 00000000000000000000000000000000 P e0ef79cd85ab4015aa59b6f1e1cd25c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 712c0ff960f04f06a6941d607ecdb2a1, e0ef79cd85ab4015aa59b6f1e1cd25c3; no voters:
I20260925 22:40:42.644953 30019 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0ef79cd85ab4015aa59b6f1e1cd25c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:40:42.645161 30019 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0ef79cd85ab4015aa59b6f1e1cd25c3 [term 1 LEADER]: Becoming Leader. State: Replica: e0ef79cd85ab4015aa59b6f1e1cd25c3, State: Running, Role: LEADER
I20260925 22:40:42.645486 30019 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0ef79cd85ab4015aa59b6f1e1cd25c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0ef79cd85ab4015aa59b6f1e1cd25c3" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 33477 } } peers { permanent_uuid: "0c238cc38fa349e5a9a3fb53e148acea" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 37165 } } peers { permanent_uuid: "712c0ff960f04f06a6941d607ecdb2a1" member_type: VOTER last_known_addr { host: "127.6.246.252" port: 46633 } }
I20260925 22:40:42.647382 30046 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0ef79cd85ab4015aa59b6f1e1cd25c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0ef79cd85ab4015aa59b6f1e1cd25c3. Latest consensus state: current_term: 1 leader_uuid: "e0ef79cd85ab4015aa59b6f1e1cd25c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0ef79cd85ab4015aa59b6f1e1cd25c3" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 33477 } } peers { permanent_uuid: "0c238cc38fa349e5a9a3fb53e148acea" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 37165 } } peers { permanent_uuid: "712c0ff960f04f06a6941d607ecdb2a1" member_type: VOTER last_known_addr { host: "127.6.246.252" port: 46633 } } }
I20260925 22:40:42.647508 30046 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0ef79cd85ab4015aa59b6f1e1cd25c3 [sys.catalog]: This master's current role is: LEADER
I20260925 22:40:42.645071 29791 master_runner.cc:186] Error getting master registration for 127.6.246.253:37165: OK, instance_id { permanent_uuid: "0c238cc38fa349e5a9a3fb53e148acea" instance_seqno: 1790376042354983 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 22:40:42.648396 29791 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 22:40:42.651250 29934 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c238cc38fa349e5a9a3fb53e148acea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0ef79cd85ab4015aa59b6f1e1cd25c3" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 33477 } } peers { permanent_uuid: "0c238cc38fa349e5a9a3fb53e148acea" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 37165 } } peers { permanent_uuid: "712c0ff960f04f06a6941d607ecdb2a1" member_type: VOTER last_known_addr { host: "127.6.246.252" port: 46633 } }
I20260925 22:40:42.651503 29934 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c238cc38fa349e5a9a3fb53e148acea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c238cc38fa349e5a9a3fb53e148acea, State: Initialized, Role: FOLLOWER
I20260925 22:40:42.651847 29934 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c238cc38fa349e5a9a3fb53e148acea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0ef79cd85ab4015aa59b6f1e1cd25c3" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 33477 } } peers { permanent_uuid: "0c238cc38fa349e5a9a3fb53e148acea" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 37165 } } peers { permanent_uuid: "712c0ff960f04f06a6941d607ecdb2a1" member_type: VOTER last_known_addr { host: "127.6.246.252" port: 46633 } }
I20260925 22:40:42.652411 30053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c238cc38fa349e5a9a3fb53e148acea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0ef79cd85ab4015aa59b6f1e1cd25c3" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 33477 } } peers { permanent_uuid: "0c238cc38fa349e5a9a3fb53e148acea" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 37165 } } peers { permanent_uuid: "712c0ff960f04f06a6941d607ecdb2a1" member_type: VOTER last_known_addr { host: "127.6.246.252" port: 46633 } } }
I20260925 22:40:42.652592 30053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c238cc38fa349e5a9a3fb53e148acea [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:40:42.653462 29934 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c238cc38fa349e5a9a3fb53e148acea [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:40:42.653764 29908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e0ef79cd85ab4015aa59b6f1e1cd25c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0c238cc38fa349e5a9a3fb53e148acea"
I20260925 22:40:42.653990 29908 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c238cc38fa349e5a9a3fb53e148acea [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:40:42.655354 29908 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0c238cc38fa349e5a9a3fb53e148acea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0ef79cd85ab4015aa59b6f1e1cd25c3 in term 1.
W20260925 22:40:42.660810 30064 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0c238cc38fa349e5a9a3fb53e148acea: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:40:42.660899 30064 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:40:42.665489 30048 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:40:42.666590 30048 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:40:42.671490 29940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:42.661475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49226 (local address 127.6.246.253:37165)
0925 22:40:42.661608 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:42.661613 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:42.662025 (+ 412us) server_negotiation.cc:407] Connection header received
0925 22:40:42.662242 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:42.662245 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:42.662292 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:42.662355 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:42.663387 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:42.664172 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:42.665338 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:42.665666 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:42.665742 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:42.667737 (+ 1995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:42.667741 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:42.667744 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:42.667770 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:42.669535 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:42.670026 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:42.670029 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:42.670031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:42.670125 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:42.670443 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:42.670446 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:42.670448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:42.670679 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:42.670806 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:42.671166 (+ 360us) server_negotiation.cc:299] Negotiation successful
0925 22:40:42.671281 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 22:40:42.681160 30016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:42.665165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53308 (local address 127.6.246.252:46633)
0925 22:40:42.665277 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:42.665281 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:42.672338 (+ 7057us) server_negotiation.cc:407] Connection header received
0925 22:40:42.672570 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:42.672574 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:42.672626 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:42.672680 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:42.673566 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:42.674407 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:42.675494 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:42.675874 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:42.675947 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:42.677646 (+ 1699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:42.677649 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:42.677652 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:42.677680 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:42.679339 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:42.679826 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:42.679830 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:42.679832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:42.679915 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:42.680201 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:42.680205 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:42.680206 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:42.680418 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:42.680549 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:42.680882 (+ 333us) server_negotiation.cc:299] Negotiation successful
0925 22:40:42.681000 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 22:40:42.685781 29984 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 712c0ff960f04f06a6941d607ecdb2a1 [term 1 FOLLOWER]: Refusing update from remote peer e0ef79cd85ab4015aa59b6f1e1cd25c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:40:42.686676 30046 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e0ef79cd85ab4015aa59b6f1e1cd25c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "712c0ff960f04f06a6941d607ecdb2a1" member_type: VOTER last_known_addr { host: "127.6.246.252" port: 46633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:40:42.687790 29863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:42.664829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51356 (local address 127.6.246.254:33477)
0925 22:40:42.664947 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:42.664951 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:42.671448 (+ 6497us) server_negotiation.cc:407] Connection header received
0925 22:40:42.671881 (+ 433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:42.671886 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:42.671974 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:42.672057 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:42.674206 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:42.674966 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:42.681394 (+ 6428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:42.681760 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:42.681840 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:42.684119 (+ 2279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:40:42.684123 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:40:42.684126 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:40:42.684154 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:40:42.685809 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:42.686382 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:42.686386 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:42.686387 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:42.686468 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:40:42.686768 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:40:42.686771 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:40:42.686773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:40:42.686973 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:40:42.687081 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:42.687524 (+ 443us) server_negotiation.cc:299] Negotiation successful
0925 22:40:42.687641 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 22:40:42.691586 30020 sys_catalog.cc:455] T 00000000000000000000000000000000 P 712c0ff960f04f06a6941d607ecdb2a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0ef79cd85ab4015aa59b6f1e1cd25c3. Latest consensus state: current_term: 1 leader_uuid: "e0ef79cd85ab4015aa59b6f1e1cd25c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0ef79cd85ab4015aa59b6f1e1cd25c3" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 33477 } } peers { permanent_uuid: "0c238cc38fa349e5a9a3fb53e148acea" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 37165 } } peers { permanent_uuid: "712c0ff960f04f06a6941d607ecdb2a1" member_type: VOTER last_known_addr { host: "127.6.246.252" port: 46633 } } }
I20260925 22:40:42.691709 30020 sys_catalog.cc:458] T 00000000000000000000000000000000 P 712c0ff960f04f06a6941d607ecdb2a1 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:40:42.692689 29908 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0c238cc38fa349e5a9a3fb53e148acea [term 1 FOLLOWER]: Refusing update from remote peer e0ef79cd85ab4015aa59b6f1e1cd25c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:40:42.693286 30046 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e0ef79cd85ab4015aa59b6f1e1cd25c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c238cc38fa349e5a9a3fb53e148acea" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 37165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:40:42.697158 30053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c238cc38fa349e5a9a3fb53e148acea [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0ef79cd85ab4015aa59b6f1e1cd25c3. Latest consensus state: current_term: 1 leader_uuid: "e0ef79cd85ab4015aa59b6f1e1cd25c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0ef79cd85ab4015aa59b6f1e1cd25c3" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 33477 } } peers { permanent_uuid: "0c238cc38fa349e5a9a3fb53e148acea" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 37165 } } peers { permanent_uuid: "712c0ff960f04f06a6941d607ecdb2a1" member_type: VOTER last_known_addr { host: "127.6.246.252" port: 46633 } } }
I20260925 22:40:42.697302 30053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c238cc38fa349e5a9a3fb53e148acea [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:40:42.697968 30046 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0ef79cd85ab4015aa59b6f1e1cd25c3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e0ef79cd85ab4015aa59b6f1e1cd25c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0ef79cd85ab4015aa59b6f1e1cd25c3" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 33477 } } peers { permanent_uuid: "0c238cc38fa349e5a9a3fb53e148acea" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 37165 } } peers { permanent_uuid: "712c0ff960f04f06a6941d607ecdb2a1" member_type: VOTER last_known_addr { host: "127.6.246.252" port: 46633 } } }
I20260925 22:40:42.698100 30046 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0ef79cd85ab4015aa59b6f1e1cd25c3 [sys.catalog]: This master's current role is: LEADER
I20260925 22:40:42.699292 30020 sys_catalog.cc:455] T 00000000000000000000000000000000 P 712c0ff960f04f06a6941d607ecdb2a1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e0ef79cd85ab4015aa59b6f1e1cd25c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0ef79cd85ab4015aa59b6f1e1cd25c3" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 33477 } } peers { permanent_uuid: "0c238cc38fa349e5a9a3fb53e148acea" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 37165 } } peers { permanent_uuid: "712c0ff960f04f06a6941d607ecdb2a1" member_type: VOTER last_known_addr { host: "127.6.246.252" port: 46633 } } }
I20260925 22:40:42.699399 30020 sys_catalog.cc:458] T 00000000000000000000000000000000 P 712c0ff960f04f06a6941d607ecdb2a1 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:40:42.700431 30069 mvcc.cc:205] Tried to move back new op lower bound from 7333380270810222592 to 7333380270679003136. Current Snapshot: MvccSnapshot[applied={T|T < 7333380270810222592}]
I20260925 22:40:42.706387 30019 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0ef79cd85ab4015aa59b6f1e1cd25c3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e0ef79cd85ab4015aa59b6f1e1cd25c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0ef79cd85ab4015aa59b6f1e1cd25c3" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 33477 } } peers { permanent_uuid: "0c238cc38fa349e5a9a3fb53e148acea" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 37165 } } peers { permanent_uuid: "712c0ff960f04f06a6941d607ecdb2a1" member_type: VOTER last_known_addr { host: "127.6.246.252" port: 46633 } } }
I20260925 22:40:42.706518 30019 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0ef79cd85ab4015aa59b6f1e1cd25c3 [sys.catalog]: This master's current role is: LEADER
I20260925 22:40:42.707367 30053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c238cc38fa349e5a9a3fb53e148acea [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e0ef79cd85ab4015aa59b6f1e1cd25c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0ef79cd85ab4015aa59b6f1e1cd25c3" member_type: VOTER last_known_addr { host: "127.6.246.254" port: 33477 } } peers { permanent_uuid: "0c238cc38fa349e5a9a3fb53e148acea" member_type: VOTER last_known_addr { host: "127.6.246.253" port: 37165 } } peers { permanent_uuid: "712c0ff960f04f06a6941d607ecdb2a1" member_type: VOTER last_known_addr { host: "127.6.246.252" port: 46633 } } }
I20260925 22:40:42.707491 30053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c238cc38fa349e5a9a3fb53e148acea [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:40:42.708882 30048 catalog_manager.cc:1347] Generated new cluster ID: 4735f8bc8c004d8e8671af9ad5bff97a
I20260925 22:40:42.708954 30048 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:40:42.709008 30072 mvcc.cc:205] Tried to move back new op lower bound from 7333380270810222592 to 7333380270679003136. Current Snapshot: MvccSnapshot[applied={T|T < 7333380270810222592}]
I20260925 22:40:42.741887 30048 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:40:42.743181 30048 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:40:42.758478 30048 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e0ef79cd85ab4015aa59b6f1e1cd25c3: Generated new TSK 0
I20260925 22:40:42.759372 30048 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 22:40:42.788213 29863 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:40:42.784947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51366 (local address 127.6.246.254:33477)
0925 22:40:42.785082 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:42.785087 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:42.786978 (+ 1891us) server_negotiation.cc:407] Connection header received
0925 22:40:42.787255 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:42.787258 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:42.787316 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:42.787372 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:42.788060 (+ 688us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51366: BlockingRecv error: recv got EOF from 127.0.0.1:51366 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260925 22:40:42.790004 30016 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:40:42.785692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53320 (local address 127.6.246.252:46633)
0925 22:40:42.785809 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:42.785814 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:42.788649 (+ 2835us) server_negotiation.cc:407] Connection header received
0925 22:40:42.789190 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:42.789193 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:42.789251 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:42.789310 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:42.789875 (+ 565us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53320: BlockingRecv error: recv got EOF from 127.0.0.1:53320 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260925 22:40:42.791163 29940 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:40:42.786366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49240 (local address 127.6.246.253:37165)
0925 22:40:42.786491 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:42.786495 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:42.789093 (+ 2598us) server_negotiation.cc:407] Connection header received
0925 22:40:42.790358 (+ 1265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:42.790363 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:42.790431 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:42.790484 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:42.791048 (+ 564us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49240: BlockingRecv error: recv got EOF from 127.0.0.1:49240 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260925 22:40:42.795782 29940 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:40:42.794513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49254 (local address 127.6.246.253:37165)
0925 22:40:42.794639 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:42.794644 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:42.794844 (+ 200us) server_negotiation.cc:407] Connection header received
0925 22:40:42.795059 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:42.795062 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:42.795138 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:42.795189 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:42.795679 (+ 490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49254: BlockingRecv error: recv got EOF from 127.0.0.1:49254 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260925 22:40:42.800740 29940 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:40:42.799240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49270 (local address 127.6.246.253:37165)
0925 22:40:42.799338 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:42.799342 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:42.799812 (+ 470us) server_negotiation.cc:407] Connection header received
0925 22:40:42.799986 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:42.799989 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:42.800040 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:42.800088 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:42.800645 (+ 557us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49270: BlockingRecv error: recv got EOF from 127.0.0.1:49270 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260925 22:40:42.806231 29940 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:40:42.804791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49274 (local address 127.6.246.253:37165)
0925 22:40:42.804888 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:42.804892 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:42.805357 (+ 465us) server_negotiation.cc:407] Connection header received
0925 22:40:42.805543 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:42.805546 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:42.805596 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:42.805646 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:42.806137 (+ 491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49274: BlockingRecv error: recv got EOF from 127.0.0.1:49274 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260925 22:40:42.814091 29940 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:40:42.812521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49286 (local address 127.6.246.253:37165)
0925 22:40:42.812658 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:42.812662 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:42.813158 (+ 496us) server_negotiation.cc:407] Connection header received
0925 22:40:42.813338 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:42.813341 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:42.813393 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:42.813456 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:42.813993 (+ 537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49286: BlockingRecv error: recv got EOF from 127.0.0.1:49286 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260925 22:40:42.826536 29940 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:40:42.824743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49294 (local address 127.6.246.253:37165)
0925 22:40:42.824874 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:42.824878 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:42.825464 (+ 586us) server_negotiation.cc:407] Connection header received
0925 22:40:42.825684 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:42.825687 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:42.825744 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:42.825797 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:42.826381 (+ 584us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49294: BlockingRecv error: recv got EOF from 127.0.0.1:49294 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260925 22:40:42.847245 29940 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:40:42.845312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49310 (local address 127.6.246.253:37165)
0925 22:40:42.845459 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:42.845464 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:42.846112 (+ 648us) server_negotiation.cc:407] Connection header received
0925 22:40:42.846381 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:42.846387 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:42.846462 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:42.846523 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:42.847122 (+ 599us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49310: BlockingRecv error: recv got EOF from 127.0.0.1:49310 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260925 22:40:42.883225 29940 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:40:42.881941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49318 (local address 127.6.246.253:37165)
0925 22:40:42.882078 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:42.882084 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:42.882269 (+ 185us) server_negotiation.cc:407] Connection header received
0925 22:40:42.882456 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:42.882466 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:42.882518 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:42.882614 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:42.883080 (+ 466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49318: BlockingRecv error: recv got EOF from 127.0.0.1:49318 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260925 22:40:42.950384 29940 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:40:42.949413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49330 (local address 127.6.246.253:37165)
0925 22:40:42.949550 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:42.949554 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:42.949661 (+ 107us) server_negotiation.cc:407] Connection header received
0925 22:40:42.949762 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:42.949764 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:42.949814 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:42.949895 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:42.950285 (+ 390us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49330: BlockingRecv error: recv got EOF from 127.0.0.1:49330 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260925 22:40:43.081496 29940 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:40:43.080536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49342 (local address 127.6.246.253:37165)
0925 22:40:43.080667 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:43.080670 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:43.080716 (+ 46us) server_negotiation.cc:407] Connection header received
0925 22:40:43.080859 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:43.080861 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:43.080910 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:43.080993 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:43.081391 (+ 398us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49342: BlockingRecv error: recv got EOF from 127.0.0.1:49342 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260925 22:40:43.341220 29940 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:40:43.340015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49344 (local address 127.6.246.253:37165)
0925 22:40:43.340194 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:43.340198 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:43.340215 (+ 17us) server_negotiation.cc:407] Connection header received
0925 22:40:43.340373 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:43.340376 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:43.340436 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:43.340584 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:40:43.341093 (+ 509us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49344: BlockingRecv error: recv got EOF from 127.0.0.1:49344 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
I20260925 22:40:43.645638 30041 catalog_manager.cc:1259] Loaded cluster ID: 4735f8bc8c004d8e8671af9ad5bff97a
I20260925 22:40:43.645756 30041 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 712c0ff960f04f06a6941d607ecdb2a1: loading cluster ID for follower catalog manager: success
I20260925 22:40:43.647879 30041 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 712c0ff960f04f06a6941d607ecdb2a1: acquiring CA information for follower catalog manager: success
I20260925 22:40:43.649399 30041 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 712c0ff960f04f06a6941d607ecdb2a1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 22:40:43.662631 30064 catalog_manager.cc:1259] Loaded cluster ID: 4735f8bc8c004d8e8671af9ad5bff97a
I20260925 22:40:43.662748 30064 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0c238cc38fa349e5a9a3fb53e148acea: loading cluster ID for follower catalog manager: success
I20260925 22:40:43.664680 30064 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0c238cc38fa349e5a9a3fb53e148acea: acquiring CA information for follower catalog manager: success
I20260925 22:40:43.665978 30064 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0c238cc38fa349e5a9a3fb53e148acea: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 22:40:43.860742 30142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:43.856021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49354 (local address 127.6.246.253:37165)
0925 22:40:43.856328 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:43.856333 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:43.856502 (+ 169us) server_negotiation.cc:407] Connection header received
0925 22:40:43.856756 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:43.856760 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:43.856939 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:43.857088 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:40:43.857382 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:43.858051 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:43.858959 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:43.859212 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:43.859332 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:43.859694 (+ 362us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:40:43.860261 (+ 567us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:40:43.860411 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:43.860455 (+ 44us) server_negotiation.cc:299] Negotiation successful
0925 22:40:43.860551 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":113,"threads_started":1}
I20260925 22:40:43.868501 30148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:43.864508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53334 (local address 127.6.246.252:46633)
0925 22:40:43.864812 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:43.864824 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:43.864839 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:40:43.865010 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:43.865015 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:43.865262 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:43.865400 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:40:43.865604 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:43.866172 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:43.867002 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:43.867248 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:43.867325 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:43.867580 (+ 255us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:40:43.868082 (+ 502us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:40:43.868178 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:43.868267 (+ 89us) server_negotiation.cc:299] Negotiation successful
0925 22:40:43.868321 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":95,"threads_started":1}
I20260925 22:40:43.876137 30142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:43.871766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49358 (local address 127.6.246.253:37165)
0925 22:40:43.872292 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:43.872297 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:43.872310 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:40:43.872361 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:43.872364 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:43.872422 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:43.872542 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:40:43.872797 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:43.873652 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:43.874924 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:43.875275 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:43.875379 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:43.875635 (+ 256us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:40:43.875721 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:40:43.875804 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:43.875894 (+ 90us) server_negotiation.cc:299] Negotiation successful
0925 22:40:43.875957 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429}
I20260925 22:40:43.876631 30155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:43.871728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51382 (local address 127.6.246.254:33477)
0925 22:40:43.872011 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:43.872019 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:43.872048 (+ 29us) server_negotiation.cc:407] Connection header received
0925 22:40:43.872305 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:43.872315 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:43.872564 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:43.872710 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:40:43.872959 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:43.873785 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:43.874642 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:43.874963 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:43.875070 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:43.875399 (+ 329us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:40:43.875829 (+ 430us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:40:43.875904 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:43.876222 (+ 318us) server_negotiation.cc:299] Negotiation successful
0925 22:40:43.876445 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":75,"threads_started":1}
I20260925 22:40:43.877575 30148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:40:43.872308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53346 (local address 127.6.246.252:46633)
0925 22:40:43.872414 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0925 22:40:43.872417 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:40:43.872428 (+ 11us) server_negotiation.cc:407] Connection header received
0925 22:40:43.872490 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:40:43.872496 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:40:43.872548 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:40:43.872612 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:40:43.873927 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:43.874709 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:43.875779 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:40:43.876155 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:40:43.876248 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:40:43.876614 (+ 366us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:40:43.876708 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:40:43.876779 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 22:40:43.877401 (+ 622us) server_negotiation.cc:299] Negotiation successful
0925 22:40:43.877463 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 22:40:43.880195 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 29791
I20260925 22:40:43.888317 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 29866
I20260925 22:40:43.895326 7131 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskan1x8t/build/debug/bin/kudu with pid 29942
2026-09-25T22:40:43Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3105 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6393 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (291092 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST